Question for everyone : How are you using Create Text GPT in AI Builder?

[plarrue]

Hi everyone, 👋 From the AI Builder product group we want to hear from you.

👉Have you tried Text Generation in AI Builder already? How are you using it? For what use case? Is there anything you would like to see added or changed to make it even better?

👉We are interested in knowing about your AI Builder experience. Please share your story!

Thank you!

[takolota]

@JoeF-MSFT 
Did the Create text with GPT action recently get a boost / switch to a 16k token model?

I just went to test a potential scenario that would result in an error because the action response would be cut short due to exceeding the token limit, which was at around 4000 a month ago.

But I just had a 35,000 character prompt & response go through without error.

Do we know if this is expected functionality? Will it continue operating this way going forward?

This will be wonderful if it does!

[ARB_wcc]

Sir, in the era of AI, a month ago is like a year ago, why are you surprised? 😆

But I noticed it too, below is Agent 00 which is my new product manager, it has like 1k words in the initial stage of the loop and when it loads the "input" from our DB the word count is increasing to 8k / 20k on average.

The output of this agent is being used by other agents in the flow and they interact between each other in a loop that can repeat 5 times or until the final agent, which is the boss, makes the decision to end the loop.

The "boss" is reviewing inputs from all other agents in the flow and the word count is averaging from 18k to 32k

A solution like this with multiple agents (GPT connectors) interacting with each other in the same flow, would be impossible a month ago.

[Ashbhatia]

Thanks for noticing the change.

The underling model for the AI Builder was recently upgraded to use the 16K version of the model. So, makers should be able to get more tokens our of their GPT calls. This should help makers provide model detailed prompts with grounding data, few shot examples, etc. This will also enable them to get more detailed as well as simulated multi turn conversations.

I hope you all are like the change. Please let us know if there is any feedback about this, good or bad, both as welcome.

Ashish

[takolota]

@Ashbhatia 

It's awesome & just what I have been waiting for!

Now my PDF data extraction template can handle 10-20 page document prompts!

https://powerusers.microsoft.com/t5/Power-Automate-Cookbook/Extract-Data-From-PDFs-and-Images-With-G... 

[ARB_wcc]

@Ashbhatia  Can't wait for this connector to use GPT4 instead of 3.5 💪🏻🤖

You know, for the basic, run-of-the-mill data extraction, the current 3.5 version is like a trusty old steed - gets the job done without much fuss. But oh boy, when we talk about those trickier tasks needing a sprinkle of, how can I put it, "cognitive analysis", I must say GPT4 can outperform 3.5 by a significant margin. The enhanced capabilities in understanding and generating nuanced responses make it a much more robust tool, especially when it comes to handling intricate "multi-agent" scenarios.  

Below is an example with multi-agents that is yielding impressive results:

🏭 Product Description Factory Flow

In the 🏭 Product Description Factory, a symbiotic and iterative process unfolds, where specialized agents collaborate meticulously to craft product descriptions in line with Brazilian import regulations. Here's a glimpse of the orchestrated workflow:

1. 🙎🏻‍♂️ Translator Agent: Kick-starts the process by translating the initial invoice descriptions into Portuguese, morphing them into a format that sets the stage for the ensuing steps.
2. 👨🏻‍💼 Boss Agent: Functions as the central node of operations, orchestrating the flow by allocating the translated description to the appropriate Product Manager, based on the product category.
3. 👨‍💼 Product Managers: Specialized experts in various domains such as "Security", "Connectivity", and "Cabling", they collaborate closely with the Data Engineer to develop a comprehensive JSON structure, forming the backbone of the final product description.
4. 👨🏻‍🔧 Data Engineer: This agent, equipped with analytical prowess, performs fuzzy lookup to extract similar products from the database, offering a rich context that facilitates the crafting of an optimized product description.
5. 👩🏻‍💼 Product Analyst: Using the JSON blueprint as a reference, this agent meticulously constructs a detailed and regulation-compliant product description, marking a pivotal step towards final submission.
6. 🧙🏻‍♂️ Tariff Classification Specialist: Stepping in with precision, this specialist identifies the correct NCM and Class ID for the product, aligning the description with tariff classifications.
7. 👨🏻‍⚖️ Auditor Agent: In the final stretch, this agent scrutinizes the product description's quality and compliance, attributing a score that indicates the level of precision and adherence to regulations.

Upon completion of the Auditor Agent's evaluation, the feedback is channeled back to the Boss Agent. This triggers a feedback loop that can repeat up to 5 times, or until the Boss Agent deems the product description to be of satisfactory quality and precision, greenlighting it for final submission.

By harmonizing the expertise of various agents, the 🏭 Product Description Factory flow can avoid hallucination, refining the product description process to facilitate smoother customs clearance.

[takolota]

@Ashbhatia 

If they do add GPT4 as a model for this action, I hope it is something one can opt in or out of.

The less expensive GPT 3.5 Turbo is just what I need for the vast majority of data extraction scenarios.

GPT4 may only be relevant for trickier data extraction scenarios like ones that require interpreting a document’s text to determine a data value, for instance determining years of experience based on a Resume with several lines of work experience dates, but no explicit reference to the number of years a person has worked in a given type of role. For those types of scenarios I would like to allow users to switch a parameter from GPT3.5 to GPT4.

[ARB_wcc]

@takolota I completely agree 

[JoeF-MSFT]

Is anyone on this thread attending the Power Platform Conference? 🙂 https://www.powerplatformconf.com/ 

[spiik]

This is my first demo, which uses sentiment analysis, custom category recognition, entities extraction, language and phrases extraction https://www.karlex.fi/automated-email-handling-and-integration-to-public-360/

[plarrue]

Hi Karl-Johan

Thanks a lot sharing your workflow. Have you explored using the "Create Text with GPT" action, perhaps for tasks like category classification?

Regards,

Philippe

[ARB_wcc]

Hi, @plarrue. I've been pondering over something - with the "Create Text with GPT" showcasing prowess in virtually any text-based task, do we even require other connectors? 😁

To illustrate, let's delve into a snippet of our "Product Description Factory" workflow. Visualize this: a specialized connector, which we refer to as Agent 01, adeptly extracts the product description from an invoice and allocates it to the appropriate Business Unit (we have 3 prominent BUs at withing the company). Following this, Agent 02 steps in, meticulously filtering the product down to the precise category and sub-category - a feat accomplished from a vast pool of over 750 potential classifications, all through the analysis of product description alone.

Check out the animation below to get the full picture, the task is done so quickly that the SharePoint List barely have time to synchronize. (note: speed was amped up to 150% of the original pace to trim down the file size)

Product Categorization with GPT (Animation)

[ARB_wcc]

@plarrue , delving further into the topic of classification, I'd like to spotlight an instance where we leverage the GPT connector as an "Auditor Agent". In this role, it scans for "approved" product descriptions to locate specific information. Not just that, it assigns a quality rank ranging from "Bad" to "Excellent" based on the depth and accuracy of the information available. But it doesn't stop there. The connector goes a step further by furnishing "Audit Insights", offering astute recommendations to enhance the quality of descriptions. This streamlined process thus allows the team to concentrate their efforts on refining the descriptions that necessitate improvement, optimizing the workflow and enhancing the accuracy of the categorization.

[takolota]

Do we know what the requests per second / requests per minute 429 throttle limit is for Create text with GPT action / Dataverse AI Builder connection?

During testing of a data extraction on a Power App today a flow’s GPT action got a 429 error & was throttled for a couple minutes. This caused the flow to timeout before responding to the app.

This was pretty surprising considering the Dataverse connector used for the action was only called once for the OCR action & once for the GPT prompt. To the best of my knowledge there were no concurrent flows running at the time.

I’ve added more connections to the flow actions to spread the calls across more account connection instances, but I don’t know why it would ever get a 429 error with only 2 action calls on a connection.

[takolota]

This is ridiculous.

https://powerusers.microsoft.com/t5/AI-Builder/Create-Text-With-GPT-Manual-approval-action-required-...

I’m more annoyed at the precedent & mindset that would have produced such a change from Microsoft staff as it is arrogantly assuming they know more about a vast number of development scenarios than the developers who work directly with an organization’s processes themselves. This is a terrible top-down approach that ignores all the better local knowledge of developers.

Anyone who recommended or started implementing this change from warning to full requirement needs to be reined in.

[ARB_wcc]

We're currently using a preview connector, which does come with its own set of quirks 😅. But I wholeheartedly agree with your sentiments.

Additionally, one aspect I find challenging is the lack of stability in the prompts. A prompt that functions flawlessly today may not necessarily perform the same way tomorrow due to ongoing updates and changes. Implementing some form of versioning control could be a game-changer, ensuring that existing connectors remain unaffected.

[takolota]

@ARB_wcc 

I understand some may be using these actions for prompts & scenarios where there is a lot more potential for variance & so they would want to encourage oversight as much as possible.

But when I’m trying to just use this like a much much more advanced regex or text string query tool, it’s annoying to get lumped in with those other higher variance uses.

[Ashbhatia]

Hi  @takolota and @ARB_wcc ,

I understand your frustrations and concerns regarding the recent changes requiring manual approval for the "Create Text with GPT" feature in Power Automate. It's clear that these changes may seem restrictive, particularly when the intended use cases involve lower variance, like an advanced regex or text string query tool.

I want to assure you that the intention behind this requirement is not to undermine the expertise of our developer community or impose a top-down approach. Rather, it's a response to the unique challenges posed by Generative AI. These models, while incredibly advanced, sometimes have a tendency to fabricate information or are susceptible to prompt injection.  

As part of our commitment to responsible AI, we already have measures in place to moderate content and prevent the generation of inappropriate responses. However, controlling the tendency of the model to fabricate responses isn't as straightforward. Even though we guide the model to stay as factual as possible, it can sometimes deviate and produce misleading outputs. Moreover, there's also a significant risk related to 'prompt injection' in the data. Even with well-crafted prompts and automation, compromised incoming data can lead to unexpected results.

The landscape of risks associated with automation using Generative AI is a relatively new and not yet fully understood territory. As we navigate this space, we're taking a cautious approach to ensure human oversight and control over AI-generated content.

That said, we're actively committed to developing robust safety systems, and, as we gain a deeper understanding, restoring as much control as possible back to developers. Our ultimate goal is to evolve this capability collectively with you, the developer community.

Your feedback regarding the stability of prompts and the potential for versioning control is incredibly valuable. We recently upgraded to the latest gpt-3.5-turbo-16k-0613 model. Our goal to bring the latest and greatest of the GPT models to our citizen developers. The latest model also quadruples the token window from 4k to 16k. Model update might be a reason for the variance in response that you were experiencing.

Please bear in mind that this feature is still in preview. As we gather more insights and navigate this emerging field, we're continually refining and tweaking our offerings to improve the user experience. 
Your patience, understanding, and input are greatly appreciated as we work towards creating an environment that is responsive to the knowledge and feedback of developer community. Thank you for being a part of this journey with us.

 

Ashish

[ARB_wcc]

Hi @Ashbhatia 

Thank you for your thorough explanation. I appreciate the complexities involved in balancing innovation with responsible AI use, especially in a field as new and rapidly evolving as Generative AI. I'm encouraged to hear that the team is committed to improving safety measures while also aiming to restore control back to developers.

Your update on the latest GPT model and its capabilities is particularly insightful. It provides context for the changes I've noticed!

Thanks again for valuing the input from the developer community. I'm eager to see how the feature evolves and am glad to be a part of this journey 👍🏻

[martindr]

as we gain a deeper understanding, restoring as much control as possible back to developers

---

Unfortunately, the mandatory approval step has killed most of my use cases because I've been using it in synchronous rather than asynchronous scenarios.

One example: I created a conversational interface on top of a Dataverse database. The user types a question in natural language and GPT translates that into a FetchXML query which the flow then executes.  I hooked it up to a PowerApp for an interactive experience. The results are listed in a datatable and displayed on a map. GPT makes for a great new user experience.

In such scenarios the concept of an asynchronous human approval step simply doesn't fit. Having debated many GenAI things with our Legal folk, I can imagine how this is a difficult problem for the product team to solve, though.

In the meantime, I have to pause my use of the most powerful action yet. Huge punch in a small package.

[ARB_wcc]

Wow, very interesting idea! disabling static result doesn't help? Solved: Create Text With GPT - Manual approval action requ... - Power Platform Community (microsoft....