E-mail: Ivan.Maslov@UnicodeLabs.ru
Skype: MegaFinder
Facebook: https://www.facebook.com/RU.Ivan.Maslov
Facebook: https://www.facebook.com/RU.IT4Business
WhatsApp | Telegram: +7 906 722 39 25
pyOpenRPA.Robot.UIDesktop.UIOSelectorUIOActivity_Run_Dict(inUIOSelector, inActionName, inArgumentList=None, inkwArgumentObject=None)[source]¶Run the activity in UIO (UI Object)
+inUIOSelector – UIOSelector - List of items, which contains condition attributes
inActionName – UIOActivity (name) activity name string from Pywinauto
inArgumentList –
inkwArgumentObject –
pyOpenRPA.Robot.UIDesktop.UIOSelector_Exist_Bool(inUIOSelector)[source]¶pyOpenRPA.Robot.UIDesktop.UIOSelector_FocusHighlight(inUIOSelector)[source]¶Set focus and highlight (draw outline) the element (in app) by the UIO selector.
+inUIOSelector – UIOSelector - List of items, which contains condition attributes
+pyOpenRPA.Robot.UIDesktop.UIOSelector_GetChildList_UIOList(inUIOSelector=None, inBackend='win32')[source]¶Get list of child UIO’s by the parent UIOSelector
+inUIOSelector – UIOSelector - List of items, which contains condition attributes
inBackend – “win32” or “uia”
pyOpenRPA.Robot.UIDesktop.UIOSelector_Get_BitnessInt(inSpecificationList)[source]¶pyOpenRPA.Robot.UIDesktop.UIOSelector_Get_UIOActivityList(inUIOSelector)[source]¶Get the list of the UI object activities
+inUIOSelector – UIOSelector - List of items, which contains condition attributes
+pyOpenRPA.Robot.UIDesktop.UIOSelector_Get_UIOInfo(inUIOSelector)[source]¶Get the UIO dict of the attributes
+inUIOSelector – UIOSelector - List of items, which contains condition attributes
+pyOpenRPA.Robot.UIDesktop.UIOSelector_Get_UIOList(inSpecificationList, inElement=None, inFlagRaiseException=True)[source]¶pyOpenRPA.Robot.UIDesktop.UIOSelector_Highlight(inUIOSelector)[source]¶Highlight (draw outline) the element (in app) by the UIO selector.
+inUIOSelector – UIOSelector - List of items, which contains condition attributes
+pyOpenRPA.Robot.UIDesktop.UIOSelector_SafeOtherGet_Process(inUIOSelector)[source]¶pyOpenRPA.Robot.UIDesktop.UIOSelector_TryRestore_Dict(inSpecificationList)[source]¶Try to restore (maximize) window, if it’s minimized. (!IMPORTANT! When use UIA framework minimized windows doesn’t appear by the UIOSelector. You need to try restore windows and after that try to get UIO)
+inSpecificationList – UIOSelector - List of items, which contains condition attributes
+pyOpenRPA.Robot.UIDesktop.UIOSelectorsSecs_WaitAppear_List(inSpecificationListList, inWaitSecs, inFlagWaitAllInMoment=False)[source]¶UIO - UI Object (class of pywinauto UI object) [pywinauto.base_wrapper]
UIOSelector - List of dict (key attributes)
PWA - PyWinAuto
PWASpecification - List of dict (key attributes in pywinauto.find_window notation)
UIOTree - Recursive Dict of Dict … (UI Parent -> Child hierarchy)
UIOInfo - Dict of UIO attributes
UIOActivity - Activity of the UIO (UI object) from the Pywinauto module
UIOEI - UI Object info object
UIO is a User Interface Object (pyOpenRPA terminology). For maximum compatibility, this instance is inherited from the object model developed in the [pywinauto library (click to get a list of available class functions)](https://pywinauto.readthedocs.io/en/latest/code/pywinauto.base_wrapper.html).
+This approach allows us to implement useful functionality that has already been successfully developed in other libraries, and Supplement it with the missing functionality. In our case, the missing functionality is the ability to dynamically access UIO objects using UIO selectors.
+<a name=”UIOSelector_Structure_Examples”></a> +UIOSelector is the list of condition items for the UIO in GUI. Each item has condition attributes for detect applicable UIO. Here is the description of the available condition attributes in item.
+ +“depth_start” :: [int, start from 1] :: the depth index, where to start check the condition list (default 1), +“depth_end” :: [int, start from 1] :: the depth index, where to stop check the condition list (default 1), +“ctrl_index” || “index” :: [int, starts from 0] :: the index of the UIO in parent UIO child list, +“title” :: [str] :: the condition for the UIO attribute title, +“title_re” :: [str] :: regular expression (python ver) for the condition for the UIO attribute title, +“rich_text” :: [str] :: the condition for the UIO attribute rich_text, +“rich_text_re” :: [str] :: regular expression (python ver) for the condition for the UIO attribute rich_text, +“class_name” :: [str] :: the condition for the UIO attribute class_name, +“class_name_re” :: [str] :: regular expression (python ver) for the condition for the UIO attribute class_name, +“friendly_class_name” :: [str] :: the condition for the UIO attribute friendly_class_name, +“friendly_class_name_re” :: [str] :: regular expression (python ver) for the condition for the UIO attribute friendly_class_name, +“control_type” :: [str] :: the condition for the UIO attribute control_type, +“control_type_re” :: [str] :: regular expression (python ver) for the condition for the UIO attribute control_type, +“is_enabled” :: [bool] :: the condition for the UIO attribute is_enabled. If UI object is enabled on GUI, +“is_visible” :: [bool] :: the condition for the UIO attribute is_visible. If UI object is visible on GUI, +“backend” :: [str, “win32” || “uia”] :: the method of UIO extraction (default “win32”). ATTENTION! Current option can be only for the first item of the UIO selector. For the next items this option will be implemented from the first item.
+}, +{ … specification next level UIO }
+{“class_name”:”CalcFrame”, “backend”:”win32”}, # 1-st level UIO specification +{“title”:”Hex”, “depth_start”:3, “depth_end”: 3} # 3-rd level specification (because of attribute depth_start|depth_stop)
+The UIDesktop is extension of the pywinauto module which provide access to the desktop apps by the win32 and ui automation dll frameworks (big thx to the Microsoft :) ).
# EXAMPLE 1
from pyOpenRPA.Robot import UIDesktop
@@ -217,21 +292,39 @@
Wait for UI object will disappear in GUI for inWaitSecs seconds.
UIOSelector_Exist_Bool(inUIOSelector)
UIOSelectorUIOActivity_Run_Dict(…[, …])
Run the activity in UIO (UI Object)
UIOSelector_Exist_Bool(inUIOSelector)
Check if object is exist by the UIO selector.
UIOSelector_Get_BitnessInt(inSpecificationList)
UIOSelector_FocusHighlight(inUIOSelector)
Set focus and highlight (draw outline) the element (in app) by the UIO selector.
Get list of child UIO’s by the parent UIOSelector
UIOSelector_Get_BitnessInt(inSpecificationList)
Detect process bitness by the UI Object UIO Selector.
UIOSelector_Get_BitnessStr(inSpecificationList)
UIOSelector_Get_BitnessStr(inSpecificationList)
Detect process bitness by the UI Object UIO Selector.
UIOSelector_Get_UIO(inSpecificationList[, …])
UIOSelector_Get_UIO(inSpecificationList[, …])
Get the pywinauto object by the UIO selector.
UIOSelector_Get_UIOList(inSpecificationList)
UIOSelector_Get_UIOActivityList(inUIOSelector)
Get the list of the UI object activities
UIOSelector_Get_UIOInfo(inUIOSelector)
Get the UIO dict of the attributes
UIOSelector_Get_UIOList(inSpecificationList)
Get the UIO list by the selector
UIOSelector_Highlight(inUIOSelector)
Highlight (draw outline) the element (in app) by the UIO selector.
UIOSelector_SafeOtherGet_Process(inUIOSelector)
Safe get other process or None if destination app is the other/same bitness
!!!!Safe call is included (you can set activity and UIDesktop will choose the bitness and return the result)!!!!!
UIOSelectorsSecs_WaitAppear_List(…[, …])
UIOSelector_TryRestore_Dict(inSpecificationList)
Try to restore (maximize) window, if it’s minimized.
UIOSelectorsSecs_WaitAppear_List(…[, …])
Wait for many UI object will appear in GUI for inWaitSecs seconds.
Wait for many UI object will disappear in GUI for inWaitSecs seconds.
pyOpenRPA.Robot.UIDesktop.UIOSelectorUIOActivity_Run_Dict(inUIOSelector, inActionName, inArgumentList=None, inkwArgumentObject=None)[source]¶Run the activity in UIO (UI Object)
+inUIOSelector – UIOSelector - List of items, which contains condition attributes
inActionName – UIOActivity (name) activity name string from Pywinauto
inArgumentList –
inkwArgumentObject –
pyOpenRPA.Robot.UIDesktop.UIOSelector_Exist_Bool(inUIOSelector)[source]¶pyOpenRPA.Robot.UIDesktop.UIOSelector_FocusHighlight(inUIOSelector)[source]¶Set focus and highlight (draw outline) the element (in app) by the UIO selector.
+inUIOSelector – UIOSelector - List of items, which contains condition attributes
+pyOpenRPA.Robot.UIDesktop.UIOSelector_GetChildList_UIOList(inUIOSelector=None, inBackend='win32')[source]¶Get list of child UIO’s by the parent UIOSelector
+inUIOSelector – UIOSelector - List of items, which contains condition attributes
inBackend – “win32” or “uia”
pyOpenRPA.Robot.UIDesktop.UIOSelector_Get_BitnessInt(inSpecificationList)[source]¶pyOpenRPA.Robot.UIDesktop.UIOSelector_Get_UIOActivityList(inUIOSelector)[source]¶Get the list of the UI object activities
+inUIOSelector – UIOSelector - List of items, which contains condition attributes
+pyOpenRPA.Robot.UIDesktop.UIOSelector_Get_UIOInfo(inUIOSelector)[source]¶Get the UIO dict of the attributes
+inUIOSelector – UIOSelector - List of items, which contains condition attributes
+pyOpenRPA.Robot.UIDesktop.UIOSelector_Get_UIOList(inSpecificationList, inElement=None, inFlagRaiseException=True)[source]¶pyOpenRPA.Robot.UIDesktop.UIOSelector_Highlight(inUIOSelector)[source]¶Highlight (draw outline) the element (in app) by the UIO selector.
+inUIOSelector – UIOSelector - List of items, which contains condition attributes
+pyOpenRPA.Robot.UIDesktop.UIOSelector_SafeOtherGet_Process(inUIOSelector)[source]¶pyOpenRPA.Robot.UIDesktop.UIOSelector_TryRestore_Dict(inSpecificationList)[source]¶Try to restore (maximize) window, if it’s minimized. (!IMPORTANT! When use UIA framework minimized windows doesn’t appear by the UIOSelector. You need to try restore windows and after that try to get UIO)
+inSpecificationList – UIOSelector - List of items, which contains condition attributes
+The OpenRPA is fully portable solution. It contains own python enviroment both 32 and 64 bit versions. So, you can execute your python script in several ways: - Execute in python x32 (OpenRPAResourcesWPy32-3720python-3.7.2) - Execute in python x64 (OpenRPAResourcesWPy64-3720python-3.7.2.amd64) - Execute from .cmd file
+To execute your python script in x32 bit version just write in command line from x32 python directory:
@@ -260,9 +261,8 @@import sys
sys.path.append('../../')
import GUI
@@ -270,17 +270,131 @@
import subprocess
import time
-#Highlight the UI Object in Folder explorer<br>
-GUI.UIOSelector_FocusHighlight([{"class_name":"CabinetWClass","backend":"uia"},{"ctrl_index":2},{"ctrl_index":0},{"ctrl_index":2},{"ctrl_index":0}])<br>
+#Highlight the UI Object in Folder explorer
+GUI.UIOSelector_FocusHighlight([{"class_name":"CabinetWClass","backend":"uia"},{"ctrl_index":2},{"ctrl_index":0},{"ctrl_index":2},{"ctrl_index":0}])
-#Wait 2 seconds<br>
-time.sleep(3)<br>
+#Wait 2 seconds
+time.sleep(3)
-#Loop: get child element of UI List<br>
-for lItem in GUI.UIOSelector_Get_UIO([{"class_name":"CabinetWClass","backend":"uia"},{"ctrl_index":2},{"ctrl_index":0},{"ctrl_index":2},{"ctrl_index":0}]).children():<br>
- print(str(lItem))
+#Loop: get child element of UI List
+for lItem in GUI.UIOSelector_Get_UIO([{"class_name":"CabinetWClass","backend":"uia"},{"ctrl_index":2},{"ctrl_index":0},{"ctrl_index":2},{"ctrl_index":0}]).children():
+ print(str(lItem))
Here you can find the docs and examples of the OpenRPA desktop (GUI) app access.
+UIO - UI Object (class of pywinauto UI object) [pywinauto.base_wrapper]<br> +UIOSelector - List of dict (key attributes)<br> +PWA - PyWinAuto<br> +PWASpecification - List of dict (key attributes in pywinauto.find_window notation)<br> +UIOTree - Recursive Dict of Dict … (UI Parent -> Child hierarchy)<br> +UIOInfo - Dict of UIO attributes<br> +UIOActivity - Activity of the UIO (UI object) from the Pywinauto module<br> +UIOEI - UI Object info object
+UIO is a User Interface Object (pyOpenRPA terminology). For maximum compatibility, this instance is inherited from the object model developed in the [pywinauto library (click to get a list of available class functions)](https://pywinauto.readthedocs.io/en/latest/code/pywinauto.base_wrapper.html).
+This approach allows us to implement useful functionality that has already been successfully developed in other libraries, and Supplement it with the missing functionality. In our case, the missing functionality is the ability to dynamically access UIO objects using UIO selectors.
+<a name=”UIOSelector_Structure_Examples”></a> +UIOSelector is the list of condition items for the UIO in GUI. Each item has condition attributes for detect applicable UIO. Here is the description of the available condition attributes in item.
+ +“depth_start” :: [int, start from 1] :: the depth index, where to start check the condition list (default 1), +“depth_end” :: [int, start from 1] :: the depth index, where to stop check the condition list (default 1), +“ctrl_index” || “index” :: [int, starts from 0] :: the index of the UIO in parent UIO child list, +“title” :: [str] :: the condition for the UIO attribute title, +“title_re” :: [str] :: regular expression (python ver) for the condition for the UIO attribute title, +“rich_text” :: [str] :: the condition for the UIO attribute rich_text, +“rich_text_re” :: [str] :: regular expression (python ver) for the condition for the UIO attribute rich_text, +“class_name” :: [str] :: the condition for the UIO attribute class_name, +“class_name_re” :: [str] :: regular expression (python ver) for the condition for the UIO attribute class_name, +“friendly_class_name” :: [str] :: the condition for the UIO attribute friendly_class_name, +“friendly_class_name_re” :: [str] :: regular expression (python ver) for the condition for the UIO attribute friendly_class_name, +“control_type” :: [str] :: the condition for the UIO attribute control_type, +“control_type_re” :: [str] :: regular expression (python ver) for the condition for the UIO attribute control_type, +“is_enabled” :: [bool] :: the condition for the UIO attribute is_enabled. If UI object is enabled on GUI, +“is_visible” :: [bool] :: the condition for the UIO attribute is_visible. If UI object is visible on GUI, +“backend” :: [str, “win32” || “uia”] :: the method of UIO extraction (default “win32”). ATTENTION! Current option can be only for the first item of the UIO selector. For the next items this option will be implemented from the first item.
+}, +{ … specification next level UIO }
+{“class_name”:”CalcFrame”, “backend”:”win32”}, # 1-st level UIO specification +{“title”:”Hex”, “depth_start”:3, “depth_end”: 3} # 3-rd level specification (because of attribute depth_start|depth_stop)
+The UIDesktop is extension of the pywinauto module which provide access to the desktop apps by the win32 and ui automation dll frameworks (big thx to the Microsoft :) ).
+*Naming convention: <InArgument>_<ActivityName>_<OutArgument - if exist>*<br>
+The pyOpenRPA support web app manipulation (by the Selenium lib). +More docs about selenium you can find here (https://selenium-python.readthedocs.io/)
+To start use selenium just import selenium modules in the robot tool. Here is the example of the usage.
+from selenium import webdriver
+from selenium.webdriver.common.keys import Keys
+
+driver = webdriver.Chrome()
+driver.get("http://www.python.org")
+assert "Python" in driver.title
+elem = driver.find_element_by_name("q")
+elem.clear()
+elem.send_keys("pycon")
+elem.send_keys(Keys.RETURN)
+assert "No results found." not in driver.page_source
+driver.close()
+Here you can find any ways you need to use in your business case: +- Find the exact match on the screen with the other image +- Use text recognition module (OCR) +- Use computer vision (CV) to identify the objects on screen/image/video +- Use artificial intelligence (AI) to make custom identification/classification/text recognition
+