time is funny

Did you ever wonder how the emotion hidden in words could be transferred into the one caused by a picture? How do reading impressions look like? Is there a synesthesia-like abstract connection, hidden code, thread or a string, between poetry and painting? If yes, gather around, and let me tell you more about a new project of mine, about Synesketch.

Synesketch (Synesthesia + Sketch) is a bridge between words, images, and code. It is a computer program, free software library, interactive animation, and a creative open project for textual affect sensing and creative visualization of it.

Integrating several rather divers fields – natural language processing, affective computing, algorithmic visual art and design, animation, psychology and color theory – I have constructed a Java-based toy which scans a written conversation in real time (e.g. chat session), analyses emotional manifestations, and visualizes it via interactive algorithmic animated Processing graphics. Emotion types which can be recognized are the basic ones by Ekman: happiness, anger, fear, surprise, sadness, and disgust (on the picture you can see representative frames of these types, done by a demo visualization system, from left to right, top to bottom). One of the concrete Synesketch applications is the visual chat: while we talk, colors and shapes are being generated.

Besides being a new media experiment, an “automatic illuminator” with “artificial synesthesia”, Synesketch is also an open source free library, so that other Processing designers and artists can create their own text-based interactive works.

But, this is only a glimpse of the whole project. A lot, lot more on Synesketch later. For now, take a look at my thesis, where all of this is explained in detail:

http://www.krcadinac.com/papers/synesketch_rad.doc (in Serbian)

Da li ste se ikada zapitali kako se emocija skrivena u rečima može pretvoriti u onu izazvanu slikom? Kako doživljaju tokom čitanja zaista izgledaju? Postoji li apstraktna veza nalik na sinesteziju, skriveni kod, između poezije i slikarstva? Ako da, poslušajte nekoliko reči o mom novom projektu, projektu Synesketch.

Synesketch (Synesthesia + Sketch) je most između reči, slike i koda. To je kompjuterski program, softverska biblioteka, interaktivna animacija i kreativni otvoreni projekat za prepoznavanje i vizuelizaciju emotivnog sadržaja teksta.

Integrišući nekoliko solidno udaljenih oblasti – procesiranje prirodnog jezika, afektivno računarstvo, algoritamsku vizuelnu umetnost i dizajn, animaciju, psihologiju i teoriju boja – napravio sam Java igračku koja čita tekstualne razgovore u realnom vremenu (npr. čet sesije), prepoznaje manifestacije emocija i vizuelizuje ih interaktivnom algoritamskom animiranom Processing grafikom. Tipovi emocija koji mogu biti prepoznati su osnovni Ekmanovi: radost, bes, strah, čuđenje, tuga, i gađenje (na slici se vide reprezentativni frejmovi ovih tipova generisani od strane demonstracionog sistema za vizuelizaciju, s leva na desno, od gore ka dole). Jedna od konkretnih Synesketch aplikacija je vizuelni čet: dok mi razgovaramo, boje i oblici se generišu u obližnjem prozoru.

Pored eksperimenta u oblasti novih medija, „automatskog iluminatora” sa „veštačkom sinestezijom”, Synesketch je takođe i open source besplatna biblioteka kojom svaki Processing dizajner i umetnik može praviti svoje interaktivne radove bazirane na tekstu.

No, ovo je samo najava. Mnogo, mnogo više o Synesketchu kasnije. Za sada, možete pogledati rad u kome je ovo opisano u detalje:

http://www.krcadinac.com/papers/synesketch_rad.doc

In the just released Encyclopedia of Information Communication Technology, edited by Antonio Cartelli, you can find an article of Milan and me about software agents, written more then a year ago. (See the list of contributors.)

ASCII portrait of my J.

  ###
 #  m
# (@«
#   ¬{})
# (@«
 #  w*
  ###

“Code is poetry.”

– wordpress.org

Our frisbee club has a good old tradition of writing frisbee-philosophy oriented haiku poems. (See one of the earlier posts, in Serbian.) But, using words is just one way to write it. You can also draw, film, compose or even program your haikus. Draw and film OK, you say, but program – it just sounds weird. Well, wait till you hear about Processing, a beautiful programming tool for visual artists and designers. But first, let me say something about these little drawings in grayscale, my visual frisbee haiku.

Seeing your dear flying disc getting entangled in the branches and leaves can be painful sometimes. Yet, sometimes it can strangely become a game of it’s own, enjoyable for you and hopefully the tree. Imagine your friends and you throwing discs through some old grapevine, laughing, running, climbing, again and again… I’ve tried to capture the contours of such an adventure on my first helloworldish Processing sketch, also making a haiku wordplay with an old soul classic.

And HERE it is.

As you can see, these digital drawings are computer generated: shapes and shades are nothing more then a plain old Java code. A little random number here, a little ellipse function there, and – here you have a little applet-based visual haiku poem: haikapplet! I must admit I watched it for about half an hour, being hypnotized by it’s ever-changing pictures and generative design that is always different.

Yet, this little haikapplet is nothing compared to what can you create with Processing. In short, it is a tool for programming visual art. An open project initiated by two great designers, programmers and MIT students, Ben Fry and Casey Reas. See some of the projects made by Processing here. Personally, I was always being skeptical about computer generated art – it just looked too cold, precise and non-human to me. But seeing certain personal, warm, even touching works by some brilliant artists, such as Jared Tarbell or Jonathan Harris, made me change my mind and give Processing a chance.

Besides the art making, I believe it could make a revolution in science education. Imagine elementary school girls and boys using basic physics laws and mathematical functions to create marvelous, colorful and inspirational drawings, animations, and interactive graphics! I see some pretty artworks as presents, new teenage loves and well-deserved first kisses. If teachers could see the potential of Processing and alike, in twenty years we may have more scientists and artists then Big Brother fans…

Yes, I know, but let me dream…

One of the things that occupied me in the past years was programming, and I’ll put a few words about one of the projects I’ve been working on.

TALARIA System is a computer multi-agent system created for information gathering on the Web and natural language interface research. You can see TALARIA’s homepage for more information. My friend Milan Stanković and me created it about two years ago, and it was presented at the Summer School of Multimedia Semantics ‘07 in Glasgow as one of the research projects. You can also read an article of ours that was recently published in the newsletter of the BCS Information Retrieval Specialist Group.

But besides the technical stuff, I’m full of impressions about Scotland, cold beauty queen of the World. Truly amazing she is! You can see the photo album from our trip. And if you ever get to travel there, please visit Arthur’s Seat above the Edinburgh. Watching seagulls and listening to distance echos of the city and the North sea, I danced at the top of it for an hour or so in a thrilling ecstasy of strange happiness. Later, I’ve remembered the final parts of the forth chapter of Joyce’s Portrait of the Artist as a Young Man. That wanderlusty artistic calling, an urge to create, not belong to any order, and strive for beauty… If there is a real freedom in this reality, I’ve felt a glimpse of it there. Thank you, Scotland.