I started by writing up a spec about 12 months ago. In my spare time I worked on it. I recently created a suite of custom nodes for ComfyUI:
I then updated it with a PyAutoGUI node that also does OCR. I then created a workflow converter to strip ComfyUI workflows of all the unnecessary stuff, so that Codex could create entire modules:
It probably would have been faster for me to do it all myself, but it helped me work out ways to make the original agent from this thread better. I used the cleaner/converter to get Codex to help me create the agent. I worked out the spec for it, then worked through module after module to get it made.
I haven’t had time to test it yet, but I don’t want to work on this alone anymore. It’s 99% done.
It doesn’t have everything that I originally planned, but in some ways it also has more. Currently, I don’t have it dreaming. Dreaming would be braindead easy to implement if there is enough interest.
This agent has subconscious epiphanies and can imagine images for itself to think things through. It has PyAutoGUI at its disposal, and a memory-forgetting system that forgets the least important memories. It also has a subconscious emotion and role module, which injects a few simple words so the Conscious can sense the direction those are taking it.
Basically, it’s a bunch of subconscious LLMs all handling subconscious tasks to create the data for the next conscious prompt.
The two sides tend to have a conversation with each other, as if they are interacting with a human user, mainly because the models are trained that way. They are basically two LLMs talking to each other, helping each other move forward in a human-like way.
You can look over the entire workflow and files here:
If anyone wants to get in touch, you can reach me on LinkedIn:
https://www.linkedin.com/in/stevenhammon/