All posts by adamjhh

Sleepy Symbiosis

Sleepy Symbiosis

Adam Haar Horowitz, Ishaan Grover, Sophia Yang

Idea and Motivation

Edgar Allen Poe, August Kekulé, Henri Poincare, Thomas Edison, Salvador Dali: each of these thinkers regularly napped with a heavy steel ball in hand, jolted awake suddenly as they lost muscle control in Stage 2 sleep and dropped it onto the floor below. Why? To wake up and unearth creative inspiration found only in fragmented threshold consciousness, in the state of mind between drowsy Stage 1 Sleep and unconscious Stage 2. This second dream state, often overlooked, inspired everything from Surrealist painting to the discovery of Benzene’s molecular structure. Recent neuroscience has tied this semi-conscious state, called Hypnagogia, to creativity, learning, lessening of ego boundaries, fluid association of ideas, and shown it can be actively incepted with stimuli presented preceding sleep. Yet normally our hypnagogic images and ideas are entirely forgotten in the morning, when we remember snippets of different deep-sleep REM state dreams or remember nothing at all. And if we each go buy a steel ball, a simple sleep-intervention technology, we have only one option of wakeup depth, no method for inception, insight or data capture, and a system that wakes us up fully rather than maintain half-awake threshold states. We saw an opportunity with the spread of cheap consumer EEG to make a more flexible, inception programmable, data-driven system for users to direct and capture hypnagogic creativity at home. We worked with Brain Computer Interface EEG technology to create a neurofeedback-based system tracking sleep spindles—a biomarker of Stage 2 sleep—to provide real-time feedback on sleep state to users and both wake them up and record insights at adjustable thresholds of consciousness. Can integration with machines allow humans to access parts of our minds currently invisible to us? As consciousness begins to dissolve, our system kicks in to find out.

Salvador Dali’s The Persistence of Memory (1931)

 Neuroscience Background:

Creativity is an altered state of consciousness: in a moment of invention, “the creator breaks free of logic and deductive reasoning, of familiar pathways, of taken-for-granted approaches” (Khatami, 1978). The practiced pathways for cognition that structure our understanding of the world are abandoned and new, fluid associations arise. The theoretical framework for the functional neuroanatomy of altered states of consciousness generally, and creativity specifically, has often centered around deactivations in the prefrontal cortex (Dietrich, 2003; Dietrich, 2006). This center for executive, organizational cognitive function shuts down and “decrease in prefrontal activity creates less censorship from the mind, and more freely allows novel sounds and imagery to be imagined by the individual. This is flexibility,” says Dierdre Barrett of Harvard Medical School (Barrett, 1993; Barrett, 2001). These unpredictable bursts of novel insight and associations can be understood as a fragmentation of normal function, a passing breakdown of structured consciousness (Noreika, 2015). The question is, how can we cause these bursts and capture them to augment our creativity?

The same hypofrontality (frontal deactivation) underlying this flexibility is also common to the early stages of sleep that this project focuses on (Muzur, 2002). Humans go through Stage 1, 2, 3, 4, and Stage 5 (Rapid Eye Movement) sleep throughout the course of a night. Regularly remembered dreams are from REM state, but we have a second, more subtle dream-state earlier on: In the transition from wakefulness to sleep, the prefrontal cortex shuts down and cognitive alterations lead to the fragmentation of consciousness experienced as novel visual, auditory and bodily hallucinations or dreams, collectively called hypnagogia (Goupil and Bekinschtein, 2011). Hypnagogic hallucinations, linked to a decrease of theta, alpha, and beta power as measured by EEG preceding their report, can also include awareness of sleep onset (Kaplan et al., 2007), distorted perception of space (Bareham et al., 2014), and time (Minkwitz et al., 2012), as well as language alterations (Noreika, 2015). Hypofrontality thus provides an opportunity for creative idea generation in hypnagogia, if novel associations can be captured.

Image from Tagliazucchi (2013).

It comes as no surprise, then, that experiments have found evidence for a correlation between hypnagogia and enhancement of creative ability (Green, 1972; Green, et al., 1970, Green, Green, & Walters 1974; Parks, 1996; Stembridge, 1972; Whisenant & Murphy, 1977; Noreika, 2015). Hypnagogia includes illogical and fluid association of ideas, loosening of Ego Boundaries and anxiety reduction (Mavromatis 1983). Recent work has found that hypnagogia is, further, ‘inceptable’, reliably involving hypnagogic imagery related to repeated tasks and imagery in preceding awake stages. (Kussé, 2011; Stickgold, 2000) And the likelihood of thematic hypnagogia images is tied to potential learning range on a topic—suggesting hypnagogia is important for learning (Stickgold, 2000). Lastly, recent research has shown that speaking does not wake subjects out of drowsy hypnagogia, and accordingly they are able to record insights via audio (Noreika, 2015). So hypnagogia appears useful for creativity, controllable through inception, important for learning, and capturable through audio recording…this is getting interesting. How do we track it and find the right moments to intervene or record, when users are truly half asleep at threshold consciousness?

Sleep Spindles:

If we can successfully track the beginning of Stage 2, our system can wake subjects back into Stage 1 sleep using their previously recorded inception audio stimuli, containing whichever theme they want Stage 1 ideation to focus on. Sleep spindles are a reliable biomarker of the onset of Sleep Stage 2, and accordingly they give us an opportunity to track and target subject’s transition out of consciousness (Gennaro, 2002). In Sleep Stage 2, transient bursts of oscillatory activity begin in a specific range of brain frequencies, 9-15.5 hz. These rhythmic discharges are distributed across the cortex, and their sudden increase in amplitude makes them detectable by human eye or automated algorithm (Nicolas, 2001). These rhythmic bursts of synchronous excitatory post-synaptic potential reach the neocortex and are registered at the scalp on Electroencephalogram (EEG) as sleep spindles. As we are using an EEG system with electrodes concentrated over frontal lobes (the Muse EEG), we will focus on detecting the range of sleep spindles most common in frontal cortical areas, namely slow 9-13hz sleep spindles, specifically finding clearest classification in the 9-11hz range (Ujma, 2015). Since spindles orchestrate rhythmic synchrony across diffuse brain areas—synchrony being key to conscious perception binding color, space, smell and more into cohesive perceived meaningful stimuli—neuroscientists have gone so far as to posit a central role for sleep spindles in the generation and degeneration of consciousness; the thalamic dynamic core theory of consciousness (Ward, 2011). As our project focuses on explorations of threshold consciousness, the role of sleep spindles in degeneration of consciousness is key and exciting. Spindles as a tracked biomarker open up many doors for further exploration of consciousness and perception.

Opportunities to Contribute to the Field:

As far as we can tell, no system has been invented that offers neurofeedback on sleep stages. No system has been created that offers the option of inception in situ (during verified hypnagogia). No hypnagogic capture system has been invented that offers granular, data-driven, adjustable wakeup times with automatic audio capture of insights, allowing for exploration of threshold consciousness past loss of muscle control. And research into hypnagogia has been done, thus far, on astronomically expensive 32 channel EEG, effectively excluding the Maker, Hacker, and Hobbyist science and tech communities. Our system aims to fill each of these gaps using a $249.00 Muse EEG, with a fully available Developer Kit, Research Tools and Documentation free online. What follows is how we went about doing that: 

Digital Signal Processing (DSP):

Our system relies on accurate detection of sleep spindles. For the sleep spindle detection algorithm, we characterized a sleep spindle as activity in 9-11hz range that lasts for a duration of at least one second. Since we only care about detection of sleep stage 2, our loss function puts a huge penalty on false positives and a small penalty for false negatives. In other words, it is okay for the algorithm to miss sleep spindles but not okay to classify a sleep spindle and sound an alarm without the presence of one. We developed two algorithms to detect sleep stage 2, each with its own strengths and weaknesses. For the purpose of this text, we call them algorithm1 and algorithm2. Algorithm1 (Wallant, 2016) provides more predictions, has a lower precision (percentage of spindles correctly predicted) and has lesser hyperparameters. Algorithm2 provides lesser predictions, has a higher precision and more hyperparameters (Devuyst, 2011).  Since we receive data from the TP10 channel of the MUSE headband in real time as opposed to a 32 channel EEG device, our data is more noisy and requires more hyperparameters compared to existing algorithms. The DREAMS database focuses on central, fast spindles in the 12-14hz range. Because our Muse EEG has limited electrodes, we focused on frontal spindles in the 9-11hz range instead. We go into the details and preliminary results of each algorithm in the next subsection.

Algorithm 1

1)For every n seconds of data, pass the wave through a bandpass filter with 9hz as the minimum cutoff frequency and 11hz as the maximum cutoff frequency.

2)In a rolling window of 1.5 seconds, compute the rms value.

3)If rms value > threshold, return spindle detected.

Testing Algorithm1

On the dreams database, we tested the algorithm against an expert’s spindle notation with 12-14 hz as the cutoff frequency, as this series of expert notations focus on central fast spindles. Within an interval of +- 3 seconds, we got only 1 false positive for the excerpt of 108 spindles.

We further tested the algorithm on a human subject with a Muse EEG headband after changing the cutoff frequencies to 9-11 hz. The algorithm however, did not perform well owing to the noisy data from the headband. This suggested the use of another algorithm with more hyperparameters that can be tuned to each subject.

Algorithm 2

1)For every n seconds of data, pass the wave through a bandpass filter with 9hz as the minimum cutoff frequency and 11 hz as the maximum cutoff frequency. Additionally, pass the original wave through a bandpass filter with a minimum frequency of 0.5hz and maximum cutoff frequency of 40 hz.

2)Compute the rms value for each of the waves after passing the original wave through the two band-pass filters.

3)In a sliding window of 1 second, take the ratio of the rms values.

4)If ratio > threshold, add the timestamp to a list of possible spindle candidates

5)For every candidate timestamp, if there are < y candidate timestamps immediately following the candidate timestamp, the candidate timestamp is a true spindle, otherwise it is not.

The hyperparameters y and threshold are tuned for each person. After tuning the hyperparameters, the algorithm gave no false positive positives pre atonia and returned spindles post atonia. However, validation by a sleep neuroscience expert of each of our detected spindles is required to fully prove the performance of the algorithm.

Each detected sleep spindle will activate an alarm based on a previously recorded audio stimuli, to time inception to the moment of reentry into Stage 1 sleep from Stage 2.

But What If I’m Not Sleepy? Photic Entrainment and Muscle Control:

After designing our system for successful signal processing of sleep spindles, we realized that while the majority of users will make use of our system in the drowsy moments after waking up or in tired moments late at night, we should account for cases of non-drowsy users who nonetheless want to nap for generation of creative content in hypnagogia. To facilitate entry into drowsy sleep stage 1, we made use of photic and audio brainwave entrainment.

Brainwave entrainment, though not widely known, has been in use since the 1800’s. The protocol involves non-invasive stimulation of internal brain oscillations based on presentation of external oscillations users aim to engender in the brain, used both in humans and animals for research purposes (Boyden, Tsai, 2015). In short, a frequency outside the brain is meant to cause a frequency inside the brain. This effect relies on rhythmic neural responses to rhythmic stimuli, a Steady-State-Response (SSR), an effect that makes sense considering the brain responds to external stimuli with internal neural activity, and activity must be time-locked to stimuli for effective perceptual binding. Multiple studies have used rhythmic visual stimulation to drive neural oscillations in humans with a range of frequencies (Keitel, 2014). Many others have shown the presence of an auditory steady state response in humans, offering a method for exploring effective audio entrainment thought the be engendered by a Brainstem based response to environmental audio (Meltzer, 2015).

Evidence for entrainment still remains heterogeneous, with a mix of hobbyists and researchers exploring it, and difficulty separating informal and formal research. Our aim was to increase theta amplitudes (6-10hz) to ease subjects into Stage 1 Sleep, as significant increases in theta frequencies are tied to Stage 1 Sleep (Schtuze, 2015). We designed stimuli lasting 5 minutes, consisting of rhythmic alternating Black and White screen flashes and isochronic tones (audible sound emitted at regular intervals) each at 8hz. We tested this system on ourselves, presenting stimulus with eyes closed, and pulling EEG reading with Muse before and after stimulus, again with eyes closed. We saw a significant increase in theta activity (0.1169 Bels in channel AF7) as compared to time preceding stimuli (0.0341 Bels in channel AF7), and noted increases in drowsiness, thus deciding to use it as our stimulus for non-drowsy participants. The literature shows higher frequency ranges have been more thoroughly explored for entrainment than the theta we used, and shows further inquiry is needed into the effect of isochronic tones: future work will require us to test audio and visual entrainment at 8 hz separately with a far higher n of subjects. Please note: if you are going to use it below, please be aware of the risk of photosensitive epilepsy, which are seizures that occur at a rate of approximately 1.1/100,000 people when presented with flashing light stimuli (Quirk, 1995).

As often happens, in creating an effective solution to one challenge, we created another challenge for ourselves. Our entrainment stimulation created an effective significant increase in theta waves, but also created a significant increase in alpha waves. Our tuned digital signal processing for sleep spindles no longer worked, and failed even 10 minutes after stimuli, making it ineffective as a Sleep Stage tracking tool! We found no research on DSP post theta entrainment, and decided to deal with the challenge of processing in entrained states in future research, discussed below. In the meantime, for non-drowsy, entrained participants where sleep spindle detection is not possible, we created a more coarse system that sends an alarm, incepts and automatically records all based on loss of muscle control as measured by Force-Sensitive-Resistor.

Industrial Design of Mask for Photic Entrainment Delivery:

The material of the sleep mask core was chosen based on the level of comfort as well as its ability to hold phones with multiple sizes for delivering photic entrainment, alarms and audio recording. Main concerns were overall weight and cushioning ability between the human face and the rigid surface of phone. We decided on low density foam as the core material because it is light weight, it presents the right level of compression when force is applied, and it has enough structural integrity to hold the phone in place. We carved the structure of the foam to fit the human face and distribute the pressure between the face and phone. We lined the core with felt to make the surface of the mask soft to the touch. Finally, we used two elastic straps as the attachment mechanism to fit various head sizes.

 


Industrial Design of FSR Muscle Sensitive Glove:

We started the design process by looking at the most graceful and nondestructive gestures people might express when transitioning between wakefulness and sleep. Because people lose muscle control when entering stage 2 sleep, we designed a glove that sends a signal to the phone app when users are no longer able to hold a closed fist. We embedded a force sensitive resistor on the palm of the glove so users can comfortably touch the sensor during their wakeful state and send a signal to the upon sleep onset when force ceases.

 Interface Design: 

Current interface design is purely practical—buttons for recording inception, beginning photic entrainment, beginning spindle tracking, and playing past recordings. Future work will include an improved UI design

 User Experience:

Case Study:

After industrial design, we had time to test our system on one local painter, hoping to inspire surrealist hypnagogic artwork. As this was our first test, we waited for detection of multiple spindles in stage 2 to ensure avoiding false positives before Stage 2, and validated loss of muscle control in the hand (atonia 4 minutes after the subject lay down) before waking up the subject and recording insights.

Results were really exciting! We saw successfully incepted imagery (1); awareness of sleep onset (2); generation of useful, relevant ideas (3); distorted sense of space (4); illogic and fluid association of images and ideas (5). Reflections below:

I was conscious of the fact that I should observe my sleep, like aware of myself progressing into sleep (2). There was really fast images. One of them was a tiger since we talked about a tiger before (1). Some of them weren’t actual objects (5). The one I remember was a chessboard that had been compressed in the middle and stretched at the edges (4). Really quick images cycling through, and the warped, pulsating darkness pattern like when you close your eyes for a long time and you get that. And then I remember speaking to someone and the ideas were really relevant and I will use them in some essay and now I can’t remember any of it (3). It was so interesting. I remember lots of images. I really want to remember this stuff. I was convinced I was going to remember it all whilst I was sleeping because I was aware it would be useful in my life on some level, I thought I need to remember these images and this conversation they’re so interesting.

It is interesting to note that visual imagery was more effectively remembered than new linguistic generation in hypnagogia. Future work will adjust the wakeup threshold past first detected sleep spindle to optimize for recall.

After the ending of the Case Study, the artist asked to be tested on again, remarking she found the process exciting from a personal exploration and idea generation standpoint. She later sent the image below as a representation of the distorted hypnagogic chessboard image described above.

Related Work:

 Neuro:

Stickgold (2000) work on Tetris Effect, where imagery before sleep enters dreams.

Gallant (2011) reconstructing visual imagery from brain activity, opening up the possibility of externalizing and recording internal imagery.

Horikawa (2013) present machine-learning models predicting the contents of visual imagery during the sleep-onset period with 60% accuracy, given fMRI activity and self-report.

Noreika (2015) work on using speaker self report and motor movement to capture hypnagogic hallucinations, like “putting a horse into a sort of violin case” (listed below).

Tech:

Relaxation Inducing Sleep Mask (US Patent US 8852073 B2) for a light-tight sleep mask presenting audible and/or LED photoluminescent visual stimuli to promote relaxation.

Chrona: Smart pillow that tracks sleep state based on motor movement and sleep by utilizing acoustic entrainment Low-frequency sounds boost deep sleep and high-frequency sounds ease you into your day. 

Dreem – EEG headband that tracks entry into REM and uses audio entrainment to enhance the quality of deep sleep.

Future Work:

Future work should (1) attempt to solve the issue of DSP post-entrainment (2) create an algorithm which automatically tunes parameters so adjustment to specific subjects is not necessary (3) test recall improvement as spindle detection alarm thresholds change (4) continue this process with a greater n of subjects and artists who work in non-visual fields, and investigate effects of hypnagogic training over time (5) test efficacy of multiple methods of inception into hypnagogia, visual or audio, and efficacy of different types of audio alarms, namely bone conduction vs air conduction sound (6) create an AI that converses with sleepers in hypnagogia and records, vs simple alarms and recording.

 

Video Link: http://sophiayang.net/video

 

References:

DREAMS Sleep Spindle Dataset: http://www.tcts.fpms.ac.be/~devuyst/Databases/DatabaseSpindle/

Bareham, C. A., Manly, T., Pustovaya, O. V., Scott, S. K., and Bekinschtein, T. A. (2014). Losing the left side of the world: rightward shift in human spatial attention with sleep onset. Sci. Rep. 4,

Barrett, Deirdre (1993). The “committee of sleep”: A study of dream incubation for problem solving. Dreaming, Vol 3(2), Jun 1993, 115-122

Boyden, Ed. Tsai, Li-Huei (2016). Nature. Gamma frequency entrainment attenuates amyloid load and modifies microglia. 540, 230–235

Devuyst, S. (2011). Automatic Sleep Spindles Detection. 33rd Annual International Conference of the IEEE

Dietrich, A. (2006). Transient hypofrontality as a mechanism for the psychological effects of exercise. Psychiatry Research, 145 (1) (2006), pp. 79–83

De Gennaro L, Ferrara M (2003) Sleep spindles: an overview. Sleep Med Rev 7:423–440.

Goupil, L., and Bekinschtein, T. A. (2011). Cognitive processing during the transition to sleep. Arch. Ital. Biol. 150, 140–154.

Huang, TL. A comprehensive review of the psychological effects of brainwave entrainment. [Altern Ther Health Med] 2008 Sep-Oct; Vol. 14 (5), pp. 38-50.

Jankel WR, Niedermeyer E. (1985). Sleep spindles. J Clin Neurophysiol 1985; 2:1–35.

Kaplan, K. A., Itoi, A., and Dement, W. C. (2007). Awareness of sleepiness and ability to predict sleep onset: can drivers avoid falling asleep at the wheel? Sleep Med. 9, 71–79.

Keitel, Christian, Cliodhna Quigley and Philipp Ruhnau (2014). Stimulus-Driven Brain Oscillations in the Alpha Range: Entrainment of Intrinsic Rhythms or Frequency-Following Response? Journal of Neuroscience 30 July 2014, 34 (31) .

Khatami, Manoochehr (1978). Creativity and Altered States of Consciousness. Psychiatric Annals. Volume 8: Issue 3 (57-64).

Meltzer, Benjamin (2015). The steady-state response of the cerebral cortex to the beat of music reflects both the comprehension of music and attention. Frontiers in Human Neuroscience (436).

Minkwitz, J., Trenner, M. U., Sander, C., Olbrich, S., Sheldrick, A. J., Hegerl, U.,et al. (2012). Time perception at different EEG-vigilance levels. Behav. Brain Funct. 8, 50. doi: 10.1186/1744-9081-8-50

Muzur, Amir, Edward F. Pace-Schott, J.Allan Hobson (2002). The prefrontal cortex in sleep

Trends in Cognitive Sciences, Volume 6, Issue 11, 1 November 2002, Pages 475-481

Nicolas, A., Petit, D., Rompré, S. & Montplaisir, J. (2001) Sleep spindle characteristics in healthy subjects of different age groups. Clin. Neurophysiol. 112, 521–527.

Nishimoto, Gallant (2011). Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies. Current Biology Volume 21, Issue 19, 11 October 2011, Pages 1641–1646

Ogilvie, R.D., R. T. Wilkinson. (1984). The detection of sleep onset: Behavioral and physiological convergence.Psychophysiology 21, 510.

Quirk, JA. (1995). Incidence of photosensitive epilepsy: a prospective national study. Electroencephalogr Clin Neurophysiol. 1995 Oct; 95(4):260-7.

Schimicek, P., Zeitlhofer, J., Anderer, P. & Saletu, B (1994). Automatic sleep-spindle detection procedure: aspects of reliability and validity. Clin. Electroencephalogr. 25, 26–29.

Schutze, MD. (2015). The difficulty of staying awake during alpha/theta neurofeedback training. Appl Psychophysiol Biofeedback. 2015 Jun;40(2):85-94.

Stickgold, A. Malia, D. Maguire, D. Roddenberry, M. O’Connor (2000)Replaying the game: Hypnagogic images in normals and amnesics. Science 290, 350.

Tagliazucchi, E. (2013). Sleep Neuroimaging and Models of Consciousness. Frontiers in Psychology. Review.

Ujma, Peter. (2015). A comparison of two sleep spindle detection methods. Front. Hum. Neurosci., 17 February 2015

Wallant, D. C. T., Maquet, P., & Phillips, C. (2016). Sleep spindles as an electrographic element: description and automatic detection methods. Neural Plasticity.