ORPA-pyOpenRPA/Sources/GuideSphinx/Robot/03_HowToUse.rst

####################################
3. How to use
####################################

The Robot tool is the main module for production process automation. It has no graphic/console interface. All low-level actions to OS are perfoming by the Robot tool in pyOpenRPA.

**************************************************
How to execute RPA script
**************************************************
You can use the robot by the several ways:

- In Python script
- In Studio script (n/a)


Create python script
############################################
In order to use robot just add Robot tool folder in work directory and add line "import GUI" in your script.


.. code-block:: python

    import sys
    sys.path.append('../../')
    import selenium # [Web app access](https://gitlab.com/UnicodeLabs/OpenRPA/wikis/05.1.-Theory-&-practice:-Web-app-access-(Chrome,-Firefox,-Opera))<br>
    import GUI  # [Win32 & UI Automation access](https://gitlab.com/UnicodeLabs/OpenRPA/wikis/05.2.-Theory-&-practice:-Desktop-app-UI-access-(win32-and-UI-automation-dlls)) <br>
    import pyautogui #[Screen capture/recognition](https://gitlab.com/UnicodeLabs/OpenRPA/wikis/05.4.-Theory-&-practice:-Screen-capture-&-image-recognition) [#Mouse manipulation](https://gitlab.com/UnicodeLabs/OpenRPA/wikis/05.3.-Theory-&-practice:-Keyboard-&-mouse-manipulation)<br>
    import cv2 # [Computer vision](https://gitlab.com/UnicodeLabs/OpenRPA/wikis/05.4.-Theory-&-practice:-Screen-capture-&-image-recognition)<br>
    import keyboard #[Keyboard manipulation](https://gitlab.com/UnicodeLabs/OpenRPA/wikis/05.3.-Theory-&-practice:-Keyboard-&-mouse-manipulation)<br>

Execute python script
############################################

The pyOpenRPA is fully portable solution. It contains own python enviroment both 32 and 64 bit versions. So, you can execute your python script in several ways:
- Execute in python x32 (\OpenRPA\Resources\WPy32-3720\python-3.7.2)
- Execute in python x64 (\OpenRPA\Resources\WPy64-3720\python-3.7.2.amd64)
- Execute from .cmd file

Execute in the Python x32
############################################

To execute your python script in x32 bit version just write in command line from x32 python directory:

.. code-block:: python

    cd "\OpenRPA\Resources\WPy32-3720\python-3.7.2"
    python.exe "path to your python script.py"

Execute in the Python x64
############################################

To execute your python script in x32 bit version just write in command line from x32 python directory:

.. code-block:: python

    cd "\OpenRPA\Resources\WPy64-3720\python-3.7.2.amd64"
    python.exe "path to your python script.py"

Execute from .cmd file
############################################

In order to simplify the execution process you can write several code lines in file with the .cmd extension:

.. code-block:: python

    cd %~dp0
    copy /Y ..\Resources\WPy32-3720\python-3.7.2\python.exe ..\Resources\WPy32-3720\python-3.7.2\OpenRPAOrchestrator.exe
    .\..\Resources\WPy32-3720\python-3.7.2\OpenRPAOrchestrator.exe orchestratorMain.py
    pause >nul


Use in studio script (n/a)
############################################
.. code-block:: python

    import sys
    sys.path.append('../../')
    import GUI
    import keyboard
    import subprocess
    import time

    #Highlight the UI Object in Folder explorer
    GUI.UIOSelector_FocusHighlight([{"class_name":"CabinetWClass","backend":"uia"},{"ctrl_index":2},{"ctrl_index":0},{"ctrl_index":2},{"ctrl_index":0}])

    #Wait 2 seconds
    time.sleep(3)

    #Loop: get child element of UI List
    for lItem in GUI.UIOSelector_Get_UIO([{"class_name":"CabinetWClass","backend":"uia"},{"ctrl_index":2},{"ctrl_index":0},{"ctrl_index":2},{"ctrl_index":0}]).children():
        print(str(lItem))



Here you can find the docs and examples of the OpenRPA desktop (GUI) app access.

****************************************************************************************************
Theory & practice. Desktop app UI access (win32 and UI automation dlls)
****************************************************************************************************

Definitions
############################################

**UIO** - UI Object (class of pywinauto UI object) [pywinauto.base_wrapper]<br>
**UIOSelector** - List of dict (key attributes)<br>
**PWA** - PyWinAuto<br>
**PWASpecification** - List of dict (key attributes in pywinauto.find_window notation)<br>
**UIOTree** - Recursive Dict of Dict ... (UI Parent -> Child hierarchy)<br>
**UIOInfo** - Dict of UIO attributes<br>
**UIOActivity** - Activity of the UIO (UI object) from the Pywinauto module<br>
**UIOEI** - UI Object info object


What is UIO?
############################################
UIO is a User Interface Object (pyOpenRPA terminology). For maximum compatibility, this instance is inherited from the object model developed in the [pywinauto library (click to get a list of available class functions)](https://pywinauto.readthedocs.io/en/latest/code/pywinauto.base_wrapper.html).

This approach allows us to implement useful functionality that has already been successfully developed in other libraries, and Supplement it with the missing functionality. In our case, the missing functionality is the ability to dynamically access UIO objects using UIO selectors.


UIOSelector structure & example
############################################
<a name="UIOSelector_Structure_Examples"></a>
UIOSelector is the list of condition items for the UIO in GUI. Each item has condition attributes for detect applicable UIO. Here is the description of the available condition attributes in item.

**Desciption**
 ```
[
    {
        "depth_start" :: [int, start from 1] :: the depth index, where to start check the condition list (default 1),
        "depth_end" :: [int, start from 1] :: the depth index, where to stop check the condition list (default 1),
        "ctrl_index" || "index" :: [int, starts from 0] :: the index of the UIO in parent UIO child list,
        "title" :: [str] :: the condition for the UIO attribute *title*,
        "title_re" :: [str] :: regular expression (python ver) for the condition for the UIO attribute *title*,
        "rich_text" :: [str] :: the condition for the UIO attribute *rich_text*,
        "rich_text_re" :: [str] :: regular expression (python ver) for the condition for the UIO attribute *rich_text*,
        "class_name" :: [str] :: the condition for the UIO attribute *class_name*,
        "class_name_re" :: [str] :: regular expression (python ver) for the condition for the UIO attribute *class_name*,
        "friendly_class_name" :: [str] :: the condition for the UIO attribute *friendly_class_name*,
        "friendly_class_name_re" :: [str] :: regular expression (python ver) for the condition for the UIO attribute *friendly_class_name*,
        "control_type" :: [str] :: the condition for the UIO attribute *control_type*,
        "control_type_re" :: [str] :: regular expression (python ver) for the condition for the UIO attribute *control_type*,
        "is_enabled" :: [bool] :: the condition for the UIO attribute *is_enabled*. If UI object is enabled on GUI,
        "is_visible" :: [bool] :: the condition for the UIO attribute *is_visible*. If UI object is visible on GUI,
        "backend" :: [str, "win32" || "uia"] :: the method of UIO extraction (default "win32"). ATTENTION! Current option can be only for the first item of the UIO selector. For the next items this option will be implemented from the first item.
    },
    { ... specification next level UIO }
]
 ```
**The UIO selector example**
 ```
[
    {"class_name":"CalcFrame", "backend":"win32"}, # 1-st level UIO specification
    {"title":"Hex", "depth_start":3, "depth_end": 3} # 3-rd level specification (because of attribute depth_start|depth_stop)
]
 ```
The UIDesktop module (OpenRPA/Robot/UIDesktop.py)
########################################################################################
The UIDesktop is extension of the pywinauto module which provide access to the desktop apps by the **win32** and **ui automation** dll frameworks (big thx to the Microsoft :) ).

*Naming convention: \<InArgument\>\_\<ActivityName\>\_\<OutArgument - if exist>*<br>


****************************************************************************************************
Theory & practice. WEB app UI access (selenium)
****************************************************************************************************

About
###############################################
The pyOpenRPA support web app manipulation (by the Selenium lib).
More docs about selenium you can find here (https://selenium-python.readthedocs.io/)

How to use
###############################################
To start use selenium just import selenium modules in the robot tool. Here is the example of the usage.

.. code-block:: python

    from selenium import webdriver
    from selenium.webdriver.common.keys import Keys

    driver = webdriver.Chrome()
    driver.get("http://www.python.org")
    assert "Python" in driver.title
    elem = driver.find_element_by_name("q")
    elem.clear()
    elem.send_keys("pycon")
    elem.send_keys(Keys.RETURN)
    assert "No results found." not in driver.page_source
    driver.close()

****************************************************************************************************
Theory & practice. Keyboard & mouse manipulation
****************************************************************************************************


****************************************************************************************************
Theory & practice. Screen capture & image recognition
****************************************************************************************************

How to automate image recognition on PC
###########################################

Here you can find any ways you need to use in your business case:
-  Find the exact match on the screen with the other image
-  Use text recognition module (OCR)
-  Use computer vision (CV) to identify the objects on screen/image/video
-  Use artificial intelligence (AI) to make custom identification/classification/text recognition