Singularity @ ICS

External Documentation

Internal Documentation

  • Locally avaialble images
    • /home/sources/SingularityImages
    • /home/inf124/Spring2020/containers

Why Singularity

  • Security: a user in the container is the same user as the one running the container, so no privilege escalation possible.
  • Ease of deployment: There is no daemon running on root, the container is simply executed.
  • No need to mount filesystems or do bind mappings to access devices

Special Circumstances

Name Spaces

Fakeroot: If you would like to use the fakeroot feature please send email to helpdesk@ics.uci.edu requesting that your username be added to the /etc/subuid and /etc/subgid files and given a 65536 uid/gid range.

Furthermore, if the system is running CentOS7, the following option must be added to the kernel boot options: namespace.unpriv_enable=1

Usage

Module

Use the ICS module command to add singularity to your path:

module load singularity

Use the following command to have singularity added to your environment every time you login:

module initadd singularity

Shortcuts

Pulling the Laughing Cow

Load the module.

module load singularity

Pull the image:

singularity pull docker://godlovedc/lolcow

Run the image; Pull the image:

singularity run docker://godlovedc/lolcow

Running the Tensorflow Docker Container

We have been successful using the singularity program to run the Tensorflow Docker container. Use the following commands on an ICS hosts with working GPU's:

module load singularity
singularity pull docker://tensorflow/tensorflow
singularity   run --nv  docker://tensorflow/tensorflow

See Tensorflow for more infon on Tensorflow @ ICS.

Running the Postgres Docker Container

Pull the docker image:

singularity pull docker://postgres

Run the docker image with your own database files (the postgresql offered to the -B switches must be created first).

POSTGRES_PASSWORD=hansiskewl PGDATA=/var/lib/postgresql/data/pgdata singularity \
  run --writable --no-home \
  -B ~/singularity/postgres/var/lib/postgresql/data:/var/lib/postgresql/data \
  -B ~/singularity/postgres/var/run/postgresql:/var/run/postgresql  \
  -docker://postgres

You can get a default postgreseql.conf file this way:

singularity exec docker://postgres  cat /usr/share/postgresql/postgresql.conf.sample  > my-postgresql.conf

The database will be available on localhost:5432 by default. Please send mail to helpdesk@ics.uci.edu if you require ports opened for the server.

rocker/shiny Singularity Image

Docker Conversions (examples)

Tomcat Docker

Tomcat Docker Page

This is how the docker pages suggest running the tomcat:9.0 container (you won't need to do this, we're just establishing a reference point). This runs the version 9.0 of the tomcat container and maps host machine port 8888 to container port 8080.

docker run -it --rm -p 8888:8080 tomcat:9.0

Pull the docker images

cd ~/singularity
singularity pull docker://tomcat
singularity build --sandbox ics124-tomcat:simg docker://tomcat

Create your own local dirs for webapps, and if you want to change any aspect of the configuration, conf.

You are on your own to populate the webapps driectory, but you can capture the tomcat conf directory by running a shell on the systema nd using scp to copy the directory back intoy our ICS account:

[5:16:58 hans@circinus-2]singularity  shell --no-home --writable  ics124-tomcat:simg                                                                             /home/hans/singularity
Singularity ics124-tomcat:simg:/> scp -rp /usr/local/tomcat/conf openlab:/home/hans/ics124/conf
Could not create directory '/home/hans/.ssh'.
The authenticity of host 'openlab (128.195.27.41)' can't be established.
ECDSA key fingerprint is SHA256:jddej7MJbNgEm3o2IcDpg5KcVXi6Vk/KmiY4u+esPpc.
Are you sure you want to continue connecting (yes/no)? yes
Failed to add the host to the list of known hosts (/home/hans/.ssh/known_hosts).
hans@openlab's password: 
catalina.policy                                                                                                                                       100%   13KB   4.5MB/s   00:00    
catalina.properties                                                                                                                                   100% 7661     3.8MB/s   00:00    
context.xml                                                                                                                                           100% 1338   731.1KB/s   00:00    
jaspic-providers.xml                                                                                                                                  100% 1149   643.5KB/s   00:00    
jaspic-providers.xsd                                                                                                                                  100% 2313     1.1MB/s   00:00    
logging.properties                                                                                                                                    100% 3916     2.0MB/s   00:00    
server.xml                                                                                                                                            100% 7588     3.6MB/s   00:00    
tomcat-users.xml                                                                                                                                      100% 2164     1.4MB/s   00:00    
tomcat-users.xsd                                                                                                                                      100% 2633     1.6MB/s   00:00    
web.xml      

Make your changes to the configureation. In this case, change the port you are running on:

[5:18:51 hans@circinus-2]pwd                                                                                                                                     /home/hans/ics124/conf
/home/hans/ics124/conf
[5:18:58 hans@circinus-2]sed -e 's/8080/8082/g' -i server.xml  

Run with your new webapps dir and new port:

singularity  run --no-home \
  -B /home/hans/ics124/webapps:/usr/local/tomcat/webapps \
  -B /home/hans/ics124/conf:/usr/local/tomcat/conf\
  --writable  ics124-tomcat:simg

At this point I can see the service running on port 8082 on the singularity host.

MySQL Docker

MySQL Docker Page

Pull the docker images

cd ~/singularity
singularity pull docker://mysql:5.7
singularity build --sandbox ics124-mysql:simg docker://mysql:5.7

Create your own local dirs for data, and if you want to change any aspect of the configuration, conf.

Run with your new webapps dir and new port:

MYSQL_ROOT_PASSWORD=hansiskewl singularity  run --no-home \
  -B /home/hans/ics124/data:/var/lib/mysql \
  -B /home/hans/ics124/mysql:/etc/mysql/conf.d \
  --writable  ics124-mysql:simg

PHP-Apache Docker

PHP-Apache Page

docker run --name my-apache-php-app -d php:7.2-apache

Pull the docker images inside your project directory.

cd ~/singularity
singularity pull docker://php:7.2-apache
singularity build --sandbox ics124-phpapache:simg docker://php:7.2-apache

Create your own local dirs for data, and if you want to change any aspect of the configuration, conf.

Run with your new webapps dir and new port:

singularity  run --no-home \
  -B /home/hans/ics124/var/www/html:/var/www/html \
  -B /home/hans/ics124/etc/apache2:/etc/apache2 \
  --writable  ics124-phpapache:simg

Rebuilding PHP-Apache Docker with Additional Packages

PHP-Apache Page

Dockerfile

FROM php:7.3-apache
USER root
RUN apt-get update &&  apt-get install -y \
        php7.3-mysql/stable

In the same directory as your Dockerfile, run:

docker build -t inf124-php73-apache .

Pull the docker images inside your project directory.

singularity build --sandbox ics124-php73-apache:simg docker-daemon://inf124-php73-apache:latest

Create your own local dirs for data, and if you want to change any aspect of the configuration, conf.

Run with your new webapps dir and new port:

singularity  run --no-home \
  -B /home/hans/ics124/var/www/html:/var/www/html \
  -B /home/hans/ics124/etc/apache2:/etc/apache2 \
  --writable  inf124-php73-apache:simg

INF124 Spring 2021

These directions are provided as a template for inclusion in course lab documentation.

1. Ssh into Openlab

Follow these instructions on how to ssh to the openlab cluster of hosts

2a. Fetch and Edit your Configuration Files

FIRST TIME ONLY

Copy over your Spring2021 configuration folder (first time only).

cp -rp /home/inf124/Spring2021/singularity.files ~/inf124
  • Edit the ~/inf124/conf/server.xml file
    • replace the string 8082 with the tomcat port assigned to you by the instructor
    • (the number appears three times in this file).
  • Edit the ~/inf124/conf/server.xml file
    • replace the string 8005 with the tomcat shutdown port assigned to you by the instructor
  • Edit the ~/inf124/conf/tomcat-users.xmlfile
    • replace the username with your uci username, and password with a password of your choice
  • Edit the ~/inf124/mysql/my.cnf file
    • and replace the string 3307 with the mysql port assigned to you by the instructor.
  • Edit the ~/inf124/etc/apache2/ports.conf
    • and replace the string 80 and 443 with the apache ports assigned to you by the instructor.

2b. Add the singularity app to your environment:

FIRST TIME ONLY

module initadd singularity
module load singularity

3. Note your hostname

[user@circinus-14 ~]# hostname
circinus-14

4. Prepare and run your Tomcat instance:

Prepare your tomcat instances with the singularity instanced start command:

cd ~inf124/Spring2021/containers/
singularity  instance start  -B ~/inf124/var/logs:/usr/local/tomcat/logs  -B ~/inf124/webapps:/usr/local/tomcat/webapps -B ~/inf124/conf:/usr/local/tomcat/conf --writable inf124-tomcat:simg $USER-tomcat 

Execute your instance with the singularity run command shown below. Optionally, include the “&” symbol to backgound the process so you get control back on the console:

singularity run instance://$USER-tomcat &

5. Start and run your MySQL Instance:

Prepare your VM:

  • Password remember to personalize the passwords in your VM
cd ~inf124/Spring2021/containers/
MYSQL_ROOT_PASSWORD=hansiskewl singularity  instance start -B ~/inf124/data:/var/lib/mysql -B ~/inf124/mysql:/etc/mysql/conf.d -B ~/inf124/var/run:/var/run --writable inf124-mysql:simg  $USER-mysql

Execute your instance with the singularity run command shown below. Optionally, include the “&” symbol to backgound the process so you get control back on the console:

singularity run instance://$USER-mysql &

6. Start your PHP Apache Instance:

cd ~inf124/Spring2021/containers/
singularity  instance start -B ~/inf124/etc/apache2:/etc/apache2 -B ~/inf124/var/www/html:/var/www/html -B ~/inf124/usr/local/etc/php:/usr/local/etc/php -B ~/inf124/var/log:/var/log --writable  inf124-php73-apache:simg  $USER-phpapache

Execute your instance with the singularity run command shown below. Optionally, include the “&” symbol to backgound the process so you get control back on the console:

singularity run instance://$USER-phpapache

*NOTE*: For step 6 to work you must have ~/inf124/usr/local/etc/php and ~/inf124/var/log present in your home directory. Please go back and look at the new steps in step2a above if they are not preset.

7. Test Tomcat:

Test your tomcat server by connecting to it now.

  • Point your browser at http://circinus-14.ics.uci.edu:8082
    • Change circinus-14 to the appropriate host from step #2a, the one you logged into.
    • Change the 8082 number to the numbed assigned you by your instructor.
    • Add /manager/html to the end of the url, and enter your login info when prompted
http://circinus-14.ics.uci.edu:8082/manager/html

8. Test Mysql:

Connect to your mysql server:

  • Port Number: Substitute 3307 for the port number you specified in step #2a.
  • Password: Substitute the password example for the one you set when starting MySQL instance in step #5

Example:

mysql -h circinus-14 -P 3307 -u root --password=hansiskewl  

9. Test Apache:

Test your Apache server by connecting to it now.

  • Point your browser at http://circinus-14.ics.uci.edu:8080
    • Change circinus-14 to the appropriate host from step #2a, the one you logged into.
    • Change the 8080 number to the numbed assigned you by your instructor.

Stopping Instances

Use the command `singularity instance stop` to stop your instances:

singularity instance stop $USER-tomcat
singularity instance stop $USER-mysql
singularity instance stop $USER-phpapache

Other Examples

Google Brain

(base) [1:03:58 hans@arcus-13]singularity pull docker://gcr.io/deepmath/hol-light:latest                                                                                            /tmp
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:cd784148e3483c2c86c50a48e535302ab0288bebd587accf40b714fffd0646b3
 2.10 MiB / 2.10 MiB [======================================================] 0s
Copying blob sha256:95ae70b7d39f173627bc2165f9e7e3e441f950cde089e0220e5d6a0154485f04
 9.01 MiB / 9.01 MiB [======================================================] 0s
Copying blob sha256:041b6225e8dc11f4dfc0e2d7166876199e45414f7fdec3ac155131a15cf64219
 6.59 MiB / 6.59 MiB [======================================================] 0s
Copying config sha256:3575906fc4d6c3dbe993067b7fca7ff4ce14470237f597c8dd4c077171ecddcb
 1.41 KiB / 1.41 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
2020/06/15 13:04:08  info unpack layer: sha256:cd784148e3483c2c86c50a48e535302ab0288bebd587accf40b714fffd0646b3
2020/06/15 13:04:08  info unpack layer: sha256:95ae70b7d39f173627bc2165f9e7e3e441f950cde089e0220e5d6a0154485f04
2020/06/15 13:04:08  info unpack layer: sha256:041b6225e8dc11f4dfc0e2d7166876199e45414f7fdec3ac155131a15cf64219

Using Slurm with and nVidia Drivers

FAQ

There are not questions yet.

Using ICS Harbor to store images.

virtual_environments/singularity.txt · Last modified: 2021/11/19 08:12 by Hans
CC Attribution-Noncommercial-Share Alike 4.0 International
Driven by DokuWiki Recent changes RSS feed Valid CSS Valid XHTML 1.0