The force is strong here..making time to create a log is hard work as there is always so much progress being made.

But documenting what is happening is vital for those who come after us..

Every week has been significant in terms of progress and development but last week was especially pivotal.

But before I get ahead of myself let me start a bit further back, just after the beginning.

As an Artist in a land of Technology the ThoughtWorkers have made me very comfortable in this new world.

They are not only on board with the mission but are expanding upon it and taking it to new horizons, as only ThoughtWorkers know how.

Andrew McWilliams as you know is the Director of the A.I. Artist Residency came up with two brainwaves (well actually he came up with several hundred brainwaves) but two particularly interesting ones..GROUP HUDDLES & OPEN STUDIO SESSIONS .

On my other residency at TED I was fortunate enough to be invited to a special meet the Ted Head, Mr Chris Anderson, when I asked him what advice he had for us. He answered “Demons and Angels…the nature of man is either always elevating towards Angels or de-evolving towards Demons. Take time to Talk to one another as it is that spark of human connection where amazing things are born!”. I think Andrew must have been there in spirit as the GROUP HUDDLES and OPENS STUDIO SESSIONS have evolved to tap into this community of knowledge in a very unique way.

The GROUP HUDDLES is the way that the ThoughtWorkers from around the Globe who are interested in being apart of the RIOT project, connect and share ideas and develop concepts using their most valuable resources their collective minds..This process is as you would expect illuminating..We have had 3 Open Studios in the past 3 weeks, and the content has swiftly escalated from exploring various forms of new tech including Capillaries under the eyes reading your pulse and also voice recognition…all the while rebuilding the emotional engine which is the Machine Learning system.

The OPEN STUDIO is a rare occurrence in the ThoughtWorkers world where they open the doors of their coven to allow outsiders (that would be you, yes you, unless you are a baptised ThoughtWorker…

The first OPEN STUDIO last week was illuminating bringing together talented humans from the world of Art, TED, Journalism. Policy, Tech and Academia converge to explore the issues being explored through RIOT, such as bias, surveillance, ethics in A.I, behavioural psychology,etc..

The second Open Studio session was so successful in this convergence of art. Tech, and storytelling sectors. It brought Rania Amiri who creates think tanks in various conflict settings in Central and South Asia, West and the Horn of Africa, the Middle East and Europe as for the United Nations. Exploring how Digital Media could potentially resolve conflict. A Government advisor from Afghanistan exploring potential Policies which may prevent these new technologies from being exploited by regimes and governments. the curators of the Future of Storytelling Festival,  Edwina Throsby the Head of Ideas and Talks at the Sydney Opera House. exploring how digital media can be used as part of conflict resolution.

As well as international organisations joining us online from  The iconic London The National theatre immersive storytelling Studio exploring new storytelling media formats. Kinicho 3D sound designers referencing Gibson and the role of sci-fi in what we are doing.

I normally talk with these people separately having them all speaking together on one room has created an unexpected alchemy and perspective shift, as each can see the others insight.

The Open Studio was so successful that we are now creating a breakout format to continue this important dialogue.

If that wasn’t enough further Breakthroughs this week is that we finally managed to connect with two powerful partners on this quest.

Dr Hongying Meng is a senior lecturer at Brunel University London in the department of Computer Science and Engineering and an eminent expert on facial recognition, Machine Learning and Deep learning. We finally managed to connect with him and he confirmed that the path that Tanya and Angelica were pursuing was the correct direction.

Alexis from the Perception institute also joined us for a very illuminating meeting where we explored racial bias, anxiety and stereotyping and how to break the pattern. Translating this into a user experience is going to be an exciting challenge. I asked Alexis why the Perception Institute is interested in the collaboration on the riot project, she communicated that “Now before A.I becomes an established part of the Arts and Cultural sector that she would like to be at the forefront of the research.!”. VR is already in the industry and behavioural study research was done after the horse bolted. Having the full picture is going to impact chattering our journey forward.

This adventure is taking us all to unknown lands…and brining more comrades..

Just like that, the next day, I started pairing with Angelica, as our mission in the overall project became clearer: improve the accuracy of the facial emotion recognition portion of the installation, to subsequently help improve the overall experience of the installation.

Karen had collaborated with Dr. Hongying Meng and a few of his students on the prototype RIOT installation before, and we used that as a starting point. They had two approaches implemented on Matlab: one using a time-delay neural network (TDNN; architected based on this paper), and another using deep learning. We did not have the source code of these Matlab implementations, but have access to a trained model (presumably from the TDNN implementation) on Matlab.

The Big Role of Data

It soon became clear to us that, like in most machine learning projects, data plays a huge part in our success / failure. This is one of our biggest challenges – there aren’t many large datasets of facial expressions with emotion labels in the first place, and the relevant datasets we found are typically made available for research purposes. But more than that, we need data to benchmark our experiments and tweaks against the system used in the prototype, because without that, we don’t even know if we’re getting closer to achieving our goal or moving away from it. We just won’t be effective.

The dataset used for this benchmarking should also ideally be representative of the installation, as that is where we want this system to eventually be used and to perform well.

Planning the data collection and lining things up for it to happen takes a while, so Angelica and I started exploring other machine learning approaches.

Facial Emotion Recognition: State of the Art

For us as non-specialists in the field of facial emotion recognition, what’s helped us get a sense of the best-performing approaches to the task has been mostly research papers we found on arXiv, especially ones that mentioned that their approach has done well in a competition, such as Emotion in the Wild. Emotion in the Wild is particularly relevant to us as the competition entries are benchmarked against facial expressions taken in the wild (not in a lab setting). They’ve been semi-automatically extracted from movies, and therefore come in various lighting conditions, background, head poses, etc.

A technique that I think is going to be key for us is transfer learning, which allows us to take advantage of pre-trained neural network architectures that’s been proven to work well in general image classification tasks, and retrain them to classify emotions instead. An example is Inception-V3 which is easily trainable via a few scripts in TensorFlow. Keras has made it straightforward to do this with even more neural network architectures.

What Lies Ahead

Having explored state-of-the-art techniques in facial emotion recognition, what we’re preparing ourselves for is rapid experimentation, to find the one that works the best in our specific situation — the RIOT installation. As described above, determining which is best will require us to have our custom data set that is representative of the eventual RIOT environment.

With that nailed, we will experiment further with various techniques to synthetically expanding our training dataset (random scaling, crops, brightness adjustments, etc.), feature extraction, and hyperparameters of the system. Lastly, as we’re likely to come up with a system that classifies emotion based on static images but need it to eventually analyze real-time video feed, we will experiment with various techniques to smooth out our emotion predictions within a given time window.

Sentiment analysis  refers to the use of natural language processing, text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjective information.The most common of the explored and easily available are using facial, text or speech recognition. Few of the very common api can be found here.Sentimental analysis has been there for very long time in speech recognition, known as emotional analysis. It involves the measurement of pitch, shrillness,  keywords in speech, pace of speech etc as your input parameters for analyzing basic sentiments through speech and voice.

Methodologies

There are also many indirect ways adopted such as speech to text conversion, followed by the sentence tone analysis, example of which can be this.This particular version uses both the speech to text as well as the tone analyser  from IBM Watson.This particular version considers only the word used in the speech, neither the pitch or shrillness is considered.Another issues is the accuracy of Watson speech to text conversion; where as CMUSphinx  and Google speech to text api are better performer, google product even considers the dialect.There is a very crude custom built command line interface tool built by me  with google speech to text and the Watson tone analyser available here.

The problem with the approach mentioned above is that pitch is a major contributor for the expression of the emotion, which is completely lost in the speech to text conversion.The above approach can be enhanced by also recording the speech as a signal, which when further exposed to the signal processing can explain the emotion masked in the pattern of the signal.

One way to improve the above mechanism can be emotion classification of the speech by deploying the statistics on fundamental frequency, energy contour, duration of silence and voice quality.Further to improve the quality many techniques are used, for example using log frequency power coefficients, exposed to the classifiers (may be Hidden Markov, etc).Other can be training a neural network over a large database of phoneme balanced words.

Thus to get a better sentimental approach it is important to have the spectro-temporal analysis of the speech along with the phrase analysis from the speech.

Challenges

•  the real time dialogue exchange should have the correct sampling at correct time
• adjusting the power gradient as per the enviornmental noise for spectral analysis
• adjustment for the differences in the different dialect and languages
• giving correct weight to the variables if considered for multi mode approach

Conclusion

Different approaches have there own challenges, for instance the speech to text model with word extraction would not imply the right sentiment in case someone saying “oh right” may have different emotion behind them, they may be in agreement with you, or may be angry.

Why Modern Open Source Projects Fail

• Survey of 414 devs 29% response rate
• Why open source projects fail?
  • Usurped by competitor Environment 27
  • Obsolete Project 20
  • Lack of time Team 18
  • Lack of interest Team 18
  • Outdated technologies Project 14
  • Low maintainability Project 7
  • Conflicts among developers Team 3
  • Legal problems Environment 2
  • Acquisition Environment 1
• What practices impact success?
  • Contributing guidelines (large impact)
  • Continuous integration (medium impact)
  • Licences, home pages, and issue templates (small impact)
• Strategies to overcome and unmaintained account
  • Move to an org account
    • “with this kind of account it would be easier to attract new maintainers and to manage permissions.”
  • Transfer ownership [unsucccessful]
    • “We tracked the activity of the new maintainers, until February, 2017. They did not perform significant contributions to the projects, despite minor commits”
  • Accepting new core developers
    • ”volunteers offered to help with the maintenance, as core developers”
    • They were either turned away by the original owner or the owner requested a plan for development which the new core devs viewed as too high of a barrier to entry

http://opensourcesurvey.org/2017/

• Documentation helps orient newcomers: how to use a project, how to contribute back, the terms of use and contribution, and the standards of conduct in a community. Improving that documentation is an impactful way to contribute back to open source.
• When communicating on a project, use clear and accessible language for people who didn’t grow up speaking English, or read less-than-fluently.
• [My Thoughts] I wonder if, in creating the original documentation, we could 1) emphasize Karen’s vision for a tool used for the greater good and to protect the people and 2) interweave some feedback culture i.e. if you have a conflict, discuss in a private way rather than a public way. What would the downside to that be? Would it require private messaging? Is that a layer of complexity beyond? Would it be worth the investment? Could it be abused? Negative experiences impact project health
• Rudeness is the #1 negative behavior. How can that be somewhat negated upfront? State that part of the project credo is “Assume everyone is doing the best they can with what they know at that time?”
• [!!!!] The gender imbalance in open source remains profound: 95% of respondents are men; just 3% are women and 1% are non-binary.
• Half of contributors say that their open source work was somewhat or very important in getting their current role.
• Stability & security are top 2 in what users value in OS projects

https://opensource.guide/

• No matter which stage you decide to open source your project, every project should include the following documentation:
  • Open source license
  • README
  • Contributing guidelines
  • Code of conduct – https://opensource.guide/code-of-conduct/
• Name – What would it be called without the RIOT front end? http://ivantomic.com/projects/ospnc/
• How can we provide entry points for newbies to triage bugs/issues?
  • https://opensourcefriday.com/
  • https://www.codetriage.com/

image: By Bundesarchiv, Bild 102-09312 / CC-BY-SA 3.0, CC BY-SA 3.0 de, https://commons.wikimedia.org/w/index.php?curid=5414490

Part of my research is looking at what makes an open source project successful. Community has begun to emerge as a theme not only in researching open source projects, but also AI, global social unrest, and RIOT as a whole.

My initial thought on how to build a community for an open source project was really about attracting enough people. One of my ideas was to figure out who was already an important person in open source, get that person or those people on board and that would organically attract other people. Another thought I had was to market the project to people by targeting their specific interests. Taking a Netflix approach. “Hey I see you liked an open source project about facial recognition, perhaps you would like this project!”

I had the opportunity to talk to Sandra Ordonez, co-founder of the Internet Freedom Festival. I wanted to know from her what the magic formula was for building a new community. The secret, she told me, was…that…there really isn’t one. Her insight might seem obvious, but I truly thought if we target the right groups of people that would be a more efficient way of building a community. That strategy might still build a community, but it would likely be weak and subject to crumbling given the first sign of stress for individuals or small groups.

Building a strong community is an incredibly bespoke process. It is about looking at the strengths and challenges of the individuals and figuring out the best way to build relationships. It is a messy process that doesn’t have a set algorithm for success. What works in one situation may have a negative impact in another situation. What’s most important she says is, “seeing everyone as an individual and understanding they are not just a mass but a person with real feelings, emotions, goals.” People have to know each other and be invested in each other’s success in order for the project to also succeed.

While I haven’t used AIM in ages, other tech has taken its place. For example, I met one of my best friends through Foursquare. The Foursquare app (the check-in portion is now called Swarm) allowed users to create and name their own places. I created a spot for my house and named it Hall of Justice. Both being comic enthusiasts, we started chatting and became friends offline as well. Neither of us use Foursquare anymore, but our friendship has sustained. Building a strong community is a very messy and unpredictable process. The key, however, is that even if the project falls away, the connections between the people would still remain.

This was just the start of my adventure into the world of open source. This foundation of community is important because ultimately, to build the project, the community has to be happy. For example, GitHub points out the importance of documentation that is up-to-date and easy to read for both users and contributors as one quality that keeps the community happy. New questions and ponderings develop with each person I talk to and article I read. An article from Knight Lab at Northwestern University suggested having a clear vision for the project which made me wonder, what is the vision, who and what will be impacted the most by such a project? How can the vision for the project be expressed via the code of conduct and documentation? In the weeks to come, I’ll answer these questions and more in my adventures in open source.

Image: By Ryan Cash of Snowman (Ryan Cash of Snowman, via email) [CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0)%5D, via Wikimedia Commons

• Correctional Offender Management Profiling for Alternative Sanctions (Compas) AI for decision-making in judicial system – http://www.equivant.com/
  • Specifically where is it being used?

• Deep Neural Networks Can Detect Sexual Orientation from Faces – https://osf.io/fk3xr/
  • Given a single facial image, a classifier could correctly distinguish between gay and heterosexual men in 81% of cases, and in 71%of cases for women.
  • LGBT groups objected citing that this tech could be used by anti-LGBT governments and orgs to target people the algorithm identified

• http://www.pewresearch.org/fact-tank/2013/12/09/study-on-twins-suggests-our-political-beliefs-may-be-hard-wired/
  • Big 5 psychological traits:
    • Openness
    • Conscientiousness
    • Extroversion
    • Agreeableness
    • Neuroticism
      
• https://www.theguardian.com/technology/2015/jan/13/your-computer-knows-you-researchers-cambridge-stanford-university
  • “Recruiters could better match candidates with jobs based on their personality; products and services could adjust their behaviour to best match their users’ characters and changing moods.”
    
• https://www.brookings.edu/blog/techtank/2017/07/20/its-time-for-our-justice-system-to-embrace-artificial-intelligence/
  • Alleviate congestion
  • More consistent than humans
  • “But human judgment brings humans failings. Not only are there racial disparities in the sentencing process, but research suggests that extraneous factors like how recently a parole board member ate lunch or how the local college football team is doing can have significant effects on the outcome of a decision.”
    
• Adversarial examples in machine learning – https://blog.openai.com/adversarial-example-research/
  • Target autonomous vehicles by masking street signs with special paint or stickers
• This person keeps coming up: https://www.psychometrics.cam.ac.uk/about-us/directory/michal-kosinski
• Commercial facial and body recognition services – https://www.faceplusplus.com/

Thoughts…

• Do conversations and implementations of AI inherently require conversations about privacy?
• Is “outperforming humans” a good metric of success in AI?
• What’s the error threshold people are willing to tolerate?
• Humans create AI -> Is AI inherently going to have bias?  AI is inherently going to have a bias.

Image: “Lychee & Peach tea with robot.” (CC BY 2.0) by squeezeomatic

We cast the net across the universe for alchemists of agile processes, masters of dev solutions and pursuers of productisations. After rigorous recruitment and secret rights of passage the initial superheroes have been assembled. Earthlings refer to them as ThoughtWorkers…

Angelica is the foundation of the team, think of Wonder Woman but instead of an invisible lasso, code is her special weapon. Tania is the neural network creator and therefore reality definer. Steph is manifesting the ancient Manifesto of channeling the energy of the world through sacred art, but now infused with technology. Andrew as the male energy subtly guides and gently directs these powerful female forces… As for myself I am known as Karen and am merely a common artist living in an uncommon time and this is our story…