Tackling the Storage Monkey

Published on May 3, 2023 at 4:08 pm by LEW

Introduction

Some days ago I decided to try out a couple of Network Attached Storage (NAS) applications in VirtualBox before deciding which one I would like to deploy on the network. And I came to a startling conclusion. I don’t really need NAS!

As I went through the motions of setting up several different NAS solutions, the same thought kept going though my mind. What I am doing now is adequate for my current and near future needs. So why change? And more important, why do I need this?

The answer to the first question is easy, because I like to learn stuff, and I like experimenting with various computer and network configurations. So yes, I would have done the virtualization anyway. I also have access to some hardware I could use to actually deploy a NAS on my local network. And again I would do it because I enjoy the challenge.

The second question was a little harder to answer. While I was playing around with my virtual models, I keep coming back to the nagging question of how is this any better than what I am doing now. However, this is one of those dependent questions. If my workflow and usage changes in the future, I could see an actual need to deploy some type of NAS solution. For my current needs however, it is quite a bit of overkill.

I have seen a lot of Internet postings with click bate titles like “You need NAS”. While a lot of these posts provide good information, I noticed there was very little review of actual need and how that would translate to a solution. In this post I want to walk through both my experience my evaluation process of my storage needs and the conclusions I reached.

Four Types of Storage

There are basically four types of storage; storage on your computer, storage external to your computer, network storage, and backup storage. Obviously these can each be broken down into multiple types of storage. For our purpose here these are the main types or categories.

Computer Storage: This is basically your storage drive(s). Any type of storage internal to your computer. This is where the majority of your data will be at some point.

External Storage: This is any type of storage you plug into your computer. Such items as SD cards, USB dongles, or external hard drives.

Network Storage: This is storage on another computer that you reach through a network connection. NAS and File Servers (FS) fall into this category. They can be on the Local Area Network (LAN), Wide Area network (WAN) or out on the Internet some place (Drop Box, Google Drive, one-drive, etc).

Backup Storage: This can be any of the previous types of storage, except it has two specific characteristics; data redundancy, and long term storage.

Evaluating Storage Needs

Before even getting to a discussion of the benefits and liabilities of various storage solutions, we need to first get a handle on our actual needs. This basically comes down to what data we want to store, how we want to organize it, how we want to access it, and how long we need to keep it. Some of this is very obvious. But I have found in stating the obvious, to make sure and include everything, least I forget about it. So lets start with what data we might be storing, and where we might store it

How we access your files: Files stored on the local computer will always be faster to access than files stored elsewhere. And depending on your non internal storage, easier to access. In some cases this does not make much of a difference. However there are many use cases where files get written to fairly often, and where it is stored, access speed, and how the data is accessed can becomes an issue.

Operating System (OS): In general terms the operating system is stored internally and loads into memory when a computer boots. You might also have an installation image or file for your operating system, generally stored on an external device.(and if you don’t, you should).

I, as a general rule do not backup my installed operating system. I do keep an extra copies of the installation media, just in case. But that is it.

Applications: Applications can be installed/reinstalled from original installer media, or the installer can be downloaded again. In that respect they are like the OS. I keep installer images on external media (USB Key), but that is it.

Note that both OS and Applications do get updated, and in most cases I check for updates before reinstalling. This can save time and frustration with certain application that demand to be updated before they can be used.

Configuration Files: If you have customized an OS or application from the default at install, you will want to figure out where the configuration file is stored, and you will probably want to make copies.

Personal Files: This is probably the bulk of what most people need to store and back up. And to some extent this is fairly easy to figure out. Obviously pictures, videos, personal documents, projects, etc.

Some items may be a little more obscure. For example, email archives maintained by your mail client. There might be particular favorite links for your browser.

Data Pack Rat: Something else to consider, are you a data hoarder? How much of the data on your computer do you really need to keep? Everyone has different requirements and needs, but it is something you should really think about on occasion.

Active Storage vs Backup Storage: Back in the day, when internal storage was at a premium, active vs backup was a much simpler comparison. For this discussion I am defining Active Storage as where you put files you are using regularly. Backup storage is where you keep copies of all your data.

Internal/External Storage

I have three terabytes of internal M.2 storage on my current laptop. I can literally store every bit of data I have there. At one point I was using a couple of multi terabyte external drives to back up my data.

I wrote a backup script, which I ran once a week, to copy specific files and directories to an external drive. The next week I would run the backup script to another external drive (two backups separated by a week). Then I would store the original drive off site. The next week I would bring the drive back on site and create a new backup, and store the second drive off site. Then repeat the sequence every week.

Not the most robust of backup schemes, but more than a lot of other people I knew were doing. But I mention it here to highlight some principles. Because doing this did save my proverbial rear end on two occasion (one drive failure, and one theft).

The point is you need both external and internal storage. How much you need will depend on what you do with your computer, and what you want to store.

Network Storage

When I made the move to LAN storage, I built a very basic file server using samba. The smb/cifs protocol ended up being the best one to use on a mixed OS network, because while certainly not the most secure and efficient, it is the one most recognized by the largest number of OS.

With SMB it was easy to create a network folder in most file managers. Then to move files, one can simply drag and drop. Writing scripts to move files around was also fairy easy.

I eventually added NextCloud to the mix, and setup Dynamic Domain Name Service to connect from the Internet. I chose to use next cloud so that all I was exposing to the Internet was an https interface.

Since I was exposing an interface to the Internet, I added a firewall also. This was all running on an old fifth generation intel NUC with 1.25 terabytes of Internal storage.

Finally I modified my existing backup script script to run form a cron job and use rsync once a day. Once the original backup was done, rsync would only backup files that had changed. This was much more robust than my weekly backups.

One of the things I noticed was that even though I could open files on the server, there was some to a lot of lag. Upgrading network equipment compensated for some of this (I started at 10 Mb network speeds and now use multi GB network speeds). And it was not a problem in most cases. But it did become more apparent doing file intensive work, like rendering.

NAS

The reason I mention the previous information is as a comparison point. To date I have only setup small NAS systems in virtualized environments. And I want to compare this to what I am currently doing. Below are some of my findings.

Conclusion

Setting up the various NAS software available, in virtual machines, was an interesting exercise. However for the amount of data I maintain, and my work flow, it is not really worth the effort of replacing my existing file server.

Instead of setting up a permanent NAS, and since I am using about 75% of my file server storage, I will be replacing the current storage drives with larger ones. This makes sense for my situation.

Each of you should really take a serious look at your own workflow and storage needs when deciding if you should add a NAS to your network. I am willing to wager that most people do not really need a NAS type solution.

Now if you want to add one just to play with it, I am all for that. Since I have access to the hardware, I will at some point probably build a NAS, just to play with it, not because I need it.

SharePin It!

Add New Comment

Your email address will not be published. Required fields are marked *