Manually add files and folders to ownCloud

There are many ways to add files to your ownCloud:

  • FTP Upload
  • SCP Upload
  • Local disk to ownCloud disk transfer
  • USB disk to ownCloud disk transfer.

But it can be pretty slow and painful process if you are adding large number of files.

Easier way is to copy the files directly to your ownCloud data directory and scan them in order to be properly added and indexed by ownCloud.

First in order to do that you need to find the “occ” command which is usually found in the root of your owncloud web directory for example /www/owncloud/occ.

Then you copy your files and folders to the data directory where your personal files are located, it could be something like /owncloud/[USERNAME]/files/.

When your files are in the new location then you need to start the scan:

sudo -u www-data php /var/www/owncloud/occ files:scan --all

Depending on the amount of files you are adding this can take some time, and will print out the results when finished.

That’s it your files and folders are now manually added to your ownCloud.

Use wget to download entire web site for local browsing

wget \
     --recursive \
     --no-clobber \
     --page-requisites \
     --html-extension \
     --convert-links \
     --restrict-file-names=windows \
     --domains www.ffd2.com \
     --no-parent \
         http://www.ffd2.com/fridge/chacking/

This command downloads the web site http://www.ffd2.com/fridge/chacking/.

The options are:

  • –recursive: download the entire Web site.
  • –domains www.ffd2.com: don’t follow links outside www.ffd2.com.
  • –no-parent: don’t follow links outside the directory fridge/chacking/.
  • –page-requisites: get all the elements that compose the page (images, CSS and so on).
  • –html-extension: save files with the .html extension.
  • –convert-links: convert links so that they work locally, off-line.
  • –restrict-file-names=windows: modify filenames so that they will work in Windows as well.
  • –no-clobber: don’t overwrite any existing files (used in case the download is interrupted and resumed).

Make Yum use specific package version from CentOS vault

Sometimes for a specific reason you want to use certain CentOS version which is removed from the main tree. Trying to update packages will report that packages for your version do not exist, but there is a way to fix this.

If you tweak the Yum repository files and remove mirrorlist directives and enable baseurl directives instead you will be able to download packages just for that specific release.

For example replace:

#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/

With:

baseurl=http://vault.centos.org/7.3.1611/os/$basearch

Where 7.3.1611 is the specific version.

If you installed some older version of CentOS which is now moved from the main tree to the vault, you could use following script to automatically tweak all Yum repository files.

RELEASE='grep -oE '[0-9]+\.[0-9]+\.[0-9]+' /etc/centos-release'
mkdir -p /etc/yum.repos.original.d
cp -r /etc/yum.repos.d /etc/yum.repos.original.d
cd /etc/yum.repos.d
sed -i 's/^mirrorlist=/###mirrorlist=/' *
sed -i 's/^#baseurl=/baseurl=/' *
sed -i '/^baseurl/s%mirror.centos.org/centos/$releasever%vault.centos.org/$RELEASE%' *

Consistent network device naming with systemd

Traditionally network device names in linux started with eth0/wlan0 where number 0 was incremented for each additional device. Problem with this naming was that device numbers did not match actual locations in computer.

Starting with v197 systemd/udev assigns predictable, stable network interface names for all local Ethernet, WLAN and WWAN interfaces. Naming is based on firmware, topology and location information.

Advantage of this approach is that the names are fully automatic, predictable and they stay fixed even if the hardware is removed or added. This allows hardware to be replaced without network cards being renamed.

Device naming is following the schemes:

  • o = onboard device
  • s = PCI hotplug
  • x = MAC address
  • P = PCI domain
  • p = PCI bus number
  • s = slot number

For example device enp0s3 would be Ethernet device in PCI bus 0 slot 3.

Ethernet device named by MAC address would start with en followed by x and MAC address (f.e. enx00112233445566).

PCI hotplug Ethernet device would start with en followed by s and the slot number, followed by f and the function number for cards that have more than one function (f.e. ens1f4).

Openstack orchestration

Colection of Openstack resources that work together as three-tier application is called a stack. Stack uses openstack resources like instances, networks, volumes and objects, and other elements. The application runs on a stack.

Stack can be set-up via CLI or Horizon GUI which will send API request to various Openstack services (Keystone, Nova, Neutron, Cinder, Swift, Glance …).

Heat is the Openstack orchestration service that sends these API requests automatically. Stack description is taken by heat and interpreted to API requests. Heat accepts AWS Cloudformation templates written in JSON, but heat has it’s own native format called Heat Orchestration Template (HOT) based on YAML.

Create simple stack with heat orchestration template:
heat_template_version: newton

resources:
   myserver:
      type: OS::Nova::Server
      properties:
         config_drive: true
         key_name: 'erol-keypair'
         image: 'cirros-image'
         flavor: 1
         networks:
           - network: 'private_network'
Validate
openstack orchestration template validate --template simple-stack.yaml
If everything is ok, there will be no errors shown:
Description: No description
Parameters: {}
Create stack:
openstack stack create -t simple-stack.yaml simple-stack
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| id                  | c0e028fd-6d10-4bd5-b3a0-5b0509ded846 |
| stack_name          | simple-stack                         |
| description         | No description                       |
| creation_time       | 2020-04-30T13:06:57Z                 |
| updated_time        | None                                 |
| stack_status        | CREATE_IN_PROGRESS                   |
| stack_status_reason | Stack CREATE started                 |
+---------------------+--------------------------------------+
List stacks to check if it has finished creating:
openstack stack list
+--------------------------------------+--------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name   | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+--------------+-----------------+----------------------+--------------+
| c0e028fd-6d10-4bd5-b3a0-5b0509ded846 | simple-stack | CREATE_COMPLETE | 2020-04-30T13:06:57Z | None         |
+--------------------------------------+--------------+-----------------+----------------------+--------------+
List all resources belonging to the stack:
openstack stack resource list simple-stack
+---------------+--------------------------------------+------------------+-----------------+----------------------+
| resource_name | physical_resource_id                 | resource_type    | resource_status | updated_time         |
+---------------+--------------------------------------+------------------+-----------------+----------------------+
| myserver      | d6d86e6d-d904-4f90-90af-8efebf45aea2 | OS::Nova::Server | CREATE_COMPLETE | 2020-04-30T13:06:57Z |
+---------------+--------------------------------------+------------------+-----------------+----------------------+
Change server name in simple-stack.yaml:
myserver;
to:
server
Update the stack:
openstack stack update -t simple-stack.yaml.1 simple-stack*
* Changing the resource name of the server will delete old instance and create new instance with new resource_name.
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| id                  | c0e028fd-6d10-4bd5-b3a0-5b0509ded846 |
| stack_name          | simple-stack                         |
| description         | No description                       |
| creation_time       | 2020-04-30T13:06:57Z                 |
| updated_time        | 2020-04-30T13:14:59Z                 |
| stack_status        | UPDATE_IN_PROGRESS                   |
| stack_status_reason | Stack UPDATE started                 |
+---------------------+--------------------------------------+
Check if stack was updated successfuly:
openstack stack list
+--------------------------------------+--------------+-----------------+----------------------+----------------------+
| ID                                   | Stack Name   | Stack Status    | Creation Time        | Updated Time         |
+--------------------------------------+--------------+-----------------+----------------------+----------------------+
| c0e028fd-6d10-4bd5-b3a0-5b0509ded846 | simple-stack | UPDATE_COMPLETE | 2020-04-30T13:06:57Z | 2020-04-30T13:14:59Z |
+--------------------------------------+--------------+-----------------+----------------------+----------------------+

Check if resource list was updated:
openstack stack resource list simple-stack
+---------------+--------------------------------------+------------------+-----------------+----------------------+
| resource_name | physical_resource_id                 | resource_type    | resource_status | updated_time         |
+---------------+--------------------------------------+------------------+-----------------+----------------------+
| server        | 62d20902-ee6c-40a0-bdc6-e38ed81fdf7c | OS::Nova::Server | CREATE_COMPLETE | 2020-04-30T13:14:59Z |
+---------------+--------------------------------------+------------------+-----------------+----------------------+
Change network in simple-stack.yaml.1:
network: 'private_network'
to:
network: 'external_network'
openstack stack update -t simple-stack.yaml.2 simple-stack
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| id                  | c0e028fd-6d10-4bd5-b3a0-5b0509ded846 |
| stack_name          | simple-stack                         |
| description         | No description                       |
| creation_time       | 2020-04-30T13:06:57Z                 |
| updated_time        | 2020-04-30T13:39:28Z                 |
| stack_status        | UPDATE_IN_PROGRESS                   |
| stack_status_reason | Stack UPDATE started                 |
+---------------------+--------------------------------------+
openstack stack resource list simple-stack
+---------------+--------------------------------------+------------------+-----------------+----------------------+
| resource_name | physical_resource_id                 | resource_type    | resource_status | updated_time         |
+---------------+--------------------------------------+------------------+-----------------+----------------------+
| server        | 62d20902-ee6c-40a0-bdc6-e38ed81fdf7c | OS::Nova::Server | UPDATE_COMPLETE | 2020-04-30T13:39:31Z |
+---------------+--------------------------------------+------------------+-----------------+----------------------+

Retrieve all parameter functions supported by Openstack version:

openstack orchestration template version list
+--------------------------------------+------+
| version                              | type |
+--------------------------------------+------+
| AWSTemplateFormatVersion.2010-09-09  | cfn  |
| HeatTemplateFormatVersion.2012-12-12 | cfn  |
| heat_template_version.2013-05-23     | hot  |
| heat_template_version.2014-10-16     | hot  |
| heat_template_version.2015-04-30     | hot  |
| heat_template_version.2015-10-15     | hot  |
| heat_template_version.2016-04-08     | hot  |
| heat_template_version.2016-10-14     | hot  |
+--------------------------------------+------+
openstack orchestration template function list heat_template_version.2016-10-14
+-----------------+-------------------------------------------------------------------------+
| functions       | description                                                             |
+-----------------+-------------------------------------------------------------------------+
| list_join       | A function for joining one or more lists of strings.                    |
| if              | A function to return corresponding value based on condition evaluation. |
| yaql            | A function for executing a yaql expression.                             |
| digest          | A function for performing digest operations.                            |
| get_attr        | A function for resolving resource attributes.                           |
| repeat          | A function for iterating over a list of items.                          |
| resource_facade | A function for retrieving data in a parent provider template.           |
| map_replace     | A function for performing substitutions on maps.                        |
| str_replace     | A function for performing string substitutions.                         |
| get_resource    | A function for resolving resource references.                           |
| map_merge       | A function for merging maps.                                            |
| str_split       | A function for splitting delimited strings into a list.                 |
| get_param       | A function for resolving parameter references.                          |
| get_file        | A function for including a file inline.                                 |
+-----------------+-------------------------------------------------------------------------+

Parameters can be provided via command line with
--parameter imgname=cirros-image keyname=erol-keypair
Or combine several parameters under one option:
--parameter "imgname=cirros-image; keyname=erol-keypair"

If you have a lot of parameters it is probably better to put them in a file again in YAML format, with single parameters key and parameter key value pairs under that key. For example create myparameters.yaml file with following contents:

parameters:
   keyname: erol-keypair
   imgname: cirros-image

You can now use –enviroment option to include the parameter keypairs from your file to the heat orchestration template.

Contents of parameters-stack.yaml:

heat_template_version: newton

parameters:
  keyname:
    type: string
    default: erol-keypair
    description: .....
  imgname:
    type: string

resources:
   server:
      type: OS::Nova::Server
      properties:
         config_drive: true
         key_name: { get_param: keyname }
         image: { get_param: imgname }
         flavor: 1
         networks:
           - network: 'private_network'

You can create your new stack with new template and parameters file:

openstack stack create myfirststack --template parameters-stack.yaml -e myparameters.yaml
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| id                  | ccc6b0a3-f690-479b-87fd-d36b1077f53c |
| stack_name          | myfirststack                         |
| description         | No description                       |
| creation_time       | 2020-04-30T14:08:58Z                 |
| updated_time        | None                                 |
| stack_status        | CREATE_IN_PROGRESS                   |
| stack_status_reason | Stack CREATE started                 |
+---------------------+--------------------------------------+
[root@packstack ~(keystone_admin)]# openstack stack list
+--------------------------------------+--------------+-----------------+----------------------+----------------------+
| ID                                   | Stack Name   | Stack Status    | Creation Time        | Updated Time         |
+--------------------------------------+--------------+-----------------+----------------------+----------------------+
| ccc6b0a3-f690-479b-87fd-d36b1077f53c | myfirststack | CREATE_COMPLETE | 2020-04-30T14:08:58Z | None                 |
| c0e028fd-6d10-4bd5-b3a0-5b0509ded846 | simple-stack | UPDATE_COMPLETE | 2020-04-30T13:06:57Z | 2020-04-30T13:39:28Z |
+--------------------------------------+--------------+-----------------+----------------------+----------------------+
openstack stack resource list myfirststack
+---------------+--------------------------------------+------------------+-----------------+----------------------+
| resource_name | physical_resource_id                 | resource_type    | resource_status | updated_time         |
+---------------+--------------------------------------+------------------+-----------------+----------------------+
| server        | e7378fa5-ed43-48eb-b00e-17952d4aad97 | OS::Nova::Server | CREATE_COMPLETE | 2020-04-30T14:08:59Z |
+---------------+--------------------------------------+------------------+-----------------+----------------------+

Create a modified version of your HOT file called linking-stack.yaml:

heat_template_version: newton

parameters:
  keyname:
    type: string
    default: erol-keypair
    description: .....
  imgname:
    type: string

resources:
   fip:
     type: OS::Nova::FloatingIP
     properties:
       pool: external_network
   fip_assoc:
     type: OS::Nova::FloatingIPAssociation
     properties:
       server_id: { get_resource: server }
       floating_ip: { get_resource: fip }

   server:
      type: OS::Nova::Server
      properties:
         config_drive: true
         key_name: { get_param: keyname }
         image: { get_param: imgname }
         flavor: 1
         networks:
           - network: 'private_network'
outputs:
   fip_address:
      description: Floating IP address value
      value:
         get_attr: [ fip, ip ]

Create a modified version of your HOT file called linking-stack.yaml:

openstack stack create linkstack --template linking-stack.yaml -e myparameters.yaml
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| id                  | eb146b58-a00e-4d5c-897f-243d6c55c4ea |
| stack_name          | linkstack                            |
| description         | No description                       |
| creation_time       | 2020-04-30T14:22:51Z                 |
| updated_time        | None                                 |
| stack_status        | CREATE_IN_PROGRESS                   |
| stack_status_reason | Stack CREATE started                 |
+---------------------+--------------------------------------+
openstack stack list
+--------------------------------------+--------------+-----------------+----------------------+----------------------+
| ID                                   | Stack Name   | Stack Status    | Creation Time        | Updated Time         |
+--------------------------------------+--------------+-----------------+----------------------+----------------------+
| eb146b58-a00e-4d5c-897f-243d6c55c4ea | linkstack    | CREATE_COMPLETE | 2020-04-30T14:22:51Z | None                 |
| ccc6b0a3-f690-479b-87fd-d36b1077f53c | myfirststack | CREATE_COMPLETE | 2020-04-30T14:08:58Z | None                 |
| c0e028fd-6d10-4bd5-b3a0-5b0509ded846 | simple-stack | UPDATE_COMPLETE | 2020-04-30T13:06:57Z | 2020-04-30T13:39:28Z |
+--------------------------------------+--------------+-----------------+----------------------+----------------------+
openstack stack event list linkstack
2020-04-30 14:22:52Z [linkstack]: CREATE_IN_PROGRESS  Stack CREATE started
2020-04-30 14:22:54Z [linkstack.fip]: CREATE_IN_PROGRESS  state changed
2020-04-30 14:22:58Z [linkstack.link-server]: CREATE_IN_PROGRESS  state changed
2020-04-30 14:22:59Z [linkstack.fip]: CREATE_COMPLETE  state changed
2020-04-30 14:23:19Z [linkstack.link-server]: CREATE_COMPLETE  state changed
2020-04-30 14:23:24Z [linkstack.fip_assoc]: CREATE_IN_PROGRESS  state changed
2020-04-30 14:23:36Z [linkstack.fip_assoc]: CREATE_COMPLETE  state changed
2020-04-30 14:23:37Z [linkstack]: CREATE_COMPLETE  Stack CREATE completed successfully
openstack stack show linkstack
+-----------------------+----------------------------------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                                                      |
+-----------------------+----------------------------------------------------------------------------------------------------------------------------+
| id                    | eb146b58-a00e-4d5c-897f-243d6c55c4ea                                                                                       |
| stack_name            | linkstack                                                                                                                  |
| description           | No description                                                                                                             |
| creation_time         | 2020-04-30T14:22:51Z                                                                                                       |
| updated_time          | None                                                                                                                       |
| stack_status          | CREATE_COMPLETE                                                                                                            |
| stack_status_reason   | Stack CREATE completed successfully                                                                                        |
| parameters            | OS::project_id: 7f64b208cc8b4a5988317789af7f827f                                                                           |
|                       | OS::stack_id: eb146b58-a00e-4d5c-897f-243d6c55c4ea                                                                         |
|                       | OS::stack_name: linkstack                                                                                                  |
|                       | imgname: cirros-image                                                                                                      |
|                       | keyname: erol-keypair                                                                                                      |
|                       |                                                                                                                            |
| outputs               | - description: Floating IP address value                                                                                   |
|                       |   output_key: fip_address                                                                                                  |
|                       |   output_value: 172.30.152.22                                                                                              |
|                       |                                                                                                                            |
| links                 | - href: http://172.30.152.4:8004/v1/7f64b208cc8b4a5988317789af7f827f/stacks/linkstack/eb146b58-a00e-4d5c-897f-243d6c55c4ea |
|                       |   rel: self                                                                                                                |
|                       |                                                                                                                            |
| parent                | None                                                                                                                       |
| disable_rollback      | True                                                                                                                       |
| deletion_time         | None                                                                                                                       |
| stack_user_project_id | 0d97b86e82234c028d9c5f785aeeb3c9                                                                                           |
| capabilities          | []                                                                                                                         |
| notification_topics   | []                                                                                                                         |
| stack_owner           | None                                                                                                                       |
| timeout_mins          | None                                                                                                                       |
| tags                  | null                                                                                                                       |
|                       | ...                                                                                                                        |
|                       |                                                                                                                            |
+-----------------------+----------------------------------------------------------------------------------------------------------------------------+
openstack stack output list linkstack
+-------------+---------------------------+
| output_key  | description               |
+-------------+---------------------------+
| fip_address | Floating IP address value |
+-------------+---------------------------+
openstack stack output show linkstack fip_address
+--------------+---------------------------+
| Field        | Value                     |
+--------------+---------------------------+
| description  | Floating IP address value |
| output_key   | fip_address               |
| output_value | 172.30.152.22             |
+--------------+---------------------------+
openstack stack resource list linkstack
+---------------+--------------------------------------+---------------------------------+-----------------+----------------------+
| resource_name | physical_resource_id                 | resource_type                   | resource_status | updated_time         |
+---------------+--------------------------------------+---------------------------------+-----------------+----------------------+
| fip           | 732de760-891d-4684-9853-874527c8dfdf | OS::Nova::FloatingIP            | CREATE_COMPLETE | 2020-04-30T14:22:53Z |
| fip_assoc     | 4                                    | OS::Nova::FloatingIPAssociation | CREATE_COMPLETE | 2020-04-30T14:22:53Z |
| link-server   | 4174ee39-6dce-4683-8d7c-8e9a00dc9907 | OS::Nova::Server                | CREATE_COMPLETE | 2020-04-30T14:22:53Z |
+---------------+--------------------------------------+---------------------------------+-----------------+----------------------+