# Is there a way to process archived files without fully extracting them to disk?

**URL:** <https://ask.cyberinfrastructure.org/t/is-there-a-way-to-process-archived-files-without-fully-extracting-them-to-disk/264>\
**Category:** Q&A\
**Tags:** data, bash, scaling, file-formats\
**Created:** [July 6, 2018, 3:20pm UTC](https://ask.cyberinfrastructure.org/t/is-there-a-way-to-process-archived-files-without-fully-extracting-them-to-disk/264 "2018-07-06T15:20:27Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![jkingsley](https://ask.cyberinfrastructure.org/letter_avatar_proxy/v4/letter/j/ad7895/32.png) [@jkingsley](https://ask.cyberinfrastructure.org/u/jkingsley)\
**Post date:** [July 6, 2018, 3:20pm UTC](https://ask.cyberinfrastructure.org/t/is-there-a-way-to-process-archived-files-without-fully-extracting-them-to-disk/264/1 "2018-07-06T15:20:28Z")

</div>

For a particular type of analysis, I have a large set of `*.tar.gz` files. While they are relatively modest in size (100MB-1GB), they are full of roughly 1kB files. In testing, the overhead of extracting hundreds of thousands of files to disk is far more expensive than the processing I need to do on the data itself.

Is there a way of directly processing the data inside the `tar` file, without having to extract it first?

---

<div class="post-metadata">

**Author:** ![vsoch](https://ask.cyberinfrastructure.org/user_avatar/ask.cyberinfrastructure.org/vsoch/32/112_2.png) [@vsoch](https://ask.cyberinfrastructure.org/u/vsoch)\
**Post date:** [November 14, 2018, 1:36am UTC](https://ask.cyberinfrastructure.org/t/is-there-a-way-to-process-archived-files-without-fully-extracting-them-to-disk/264/7 "2018-11-14T01:36:50Z")

</div>

If you use python (or your language of choice) there is a core module called `tarfile` that can do wonders to:

1. read a tarfile into memory
2. either edit members in place and write to memory (and then update file) or write to new thing.

For example, I just wrote up this little snippet [https://gist.github.com/vsoch/5ef7cc7974d3bf94a83c16fbc8cda9a8](https://gist.github.com/vsoch/5ef7cc7974d3bf94a83c16fbc8cda9a8) to read a .tar.gz into memory, check permissions, and change if necessary. I’ll also include it here:

```python
import tarfile
import tempfile
import stat
import os

tar_file = "input.tar.gz"
tar = tarfile.open(tar_file, "r:gz")
members = tar.getmembers()

file_permission = stat.S_IRUSR | stat.S_IWUSR
folder_permission = stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR

# Let's pretend we want to edit, and write to new tar
if len(members) > 0:
    fd, tmp_tar = tempfile.mkstemp(prefix=("%s.fixed." % tar_file))
    os.close(fd)
    fixed_tar = tarfile.open(tmp_tar, "w:gz")

    # Then process members
    for member in members:

        # add o+rwx for directories
        if member.isdir() and not member.issym():
            member.mode = folder_permission | member.mode
            extracted = tar.extractfile(member)
            fixed_tar.addfile(member, extracted)

        # add o+rw for plain files
        elif member.isfile() and not member.issym():
            member.mode = file_permission | member.mode
            extracted = tar.extractfile(member)
            fixed_tar.addfile(member, extracted)
        else:
            fixed_tar.addfile(member)
            
    fixed_tar.close()
    tar.close()

    # Rename the fixed tar to be the old name
    os.rename(tmp_tar, tar_file)
else:
    tar.close()

```

That example is from the original Singularty source code, and there are other examples to:

- [Generate New Tar](https://github.com/sylabs/singularity/blob/vault/release-2.6/libexec/python/sutils.py#L153)
- [tarinfo template](https://github.com/sylabs/singularity/blob/vault/release-2.6/libexec/python/templates.py#L34)

If you have a specific need or example I’d be happy to help! We can also try outside of Python.

---

<div class="post-metadata">

**Author:** ![jkingsley](https://ask.cyberinfrastructure.org/letter_avatar_proxy/v4/letter/j/ad7895/32.png) [@jkingsley](https://ask.cyberinfrastructure.org/u/jkingsley)\
**Post date:** [July 6, 2018, 3:21pm UTC](https://ask.cyberinfrastructure.org/t/is-there-a-way-to-process-archived-files-without-fully-extracting-them-to-disk/264/2 "2018-07-06T15:21:55Z")

</div>

One approach you could take would be to extract files to a temporary location in memory. Given the size you are working with, the available space on the ram-disk that is mounted by default at `/dev/shm` should be enough, as long as you make sure to clean these files up after they are analyzed, before extracting the next set.

---

<div class="post-metadata">

**Author:** ![jpessin1](https://ask.cyberinfrastructure.org/user_avatar/ask.cyberinfrastructure.org/jpessin1/32/339_2.png) [@jpessin1](https://ask.cyberinfrastructure.org/u/jpessin1)\
**Post date:** [July 24, 2018, 1:07am UTC](https://ask.cyberinfrastructure.org/t/is-there-a-way-to-process-archived-files-without-fully-extracting-them-to-disk/264/4 "2018-07-24T01:07:46Z")

</div>

@jkingsley /dev/shm, seems like a system specific detail. Is there a more generic name for this?

---

<div class="post-metadata">

**Author:** ![jkingsley](https://ask.cyberinfrastructure.org/letter_avatar_proxy/v4/letter/j/ad7895/32.png) [@jkingsley](https://ask.cyberinfrastructure.org/u/jkingsley)\
**Post date:** [July 24, 2018, 6:11pm UTC](https://ask.cyberinfrastructure.org/t/is-there-a-way-to-process-archived-files-without-fully-extracting-them-to-disk/264/5 "2018-07-24T18:11:46Z")

</div>

`/dev/shm` has been a standard feature of linux installs for at least a decade (I don’t actually have an introduction date, but I have seen references as early as 2006). Unless it was specifically removed for some reason, I would expect it on any modern system.

---

<div class="post-metadata">

**Author:** ![jpessin1](https://ask.cyberinfrastructure.org/user_avatar/ask.cyberinfrastructure.org/jpessin1/32/339_2.png) [@jpessin1](https://ask.cyberinfrastructure.org/u/jpessin1)\
**Post date:** [July 24, 2018, 9:55pm UTC](https://ask.cyberinfrastructure.org/t/is-there-a-way-to-process-archived-files-without-fully-extracting-them-to-disk/264/6 "2018-07-24T21:55:50Z")

</div>

@jkinsley think this [https://superuser.com/questions/45342/when-should-i-use-dev-shm-and-when-should-i-use-tmp](https://superuser.com/questions/45342/when-should-i-use-dev-shm-and-when-should-i-use-tmp) covers some of that … (linux kernal 2.6) … most common linux distros do have it on by defualt, but it is an optional config.

---

<div class="post-metadata">

**Author:** ![jpessin1](https://ask.cyberinfrastructure.org/user_avatar/ask.cyberinfrastructure.org/jpessin1/32/339_2.png) [@jpessin1](https://ask.cyberinfrastructure.org/u/jpessin1)\
**Post date:** [May 31, 2019, 8:27pm UTC](https://ask.cyberinfrastructure.org/t/is-there-a-way-to-process-archived-files-without-fully-extracting-them-to-disk/264/8 "2019-05-31T20:27:49Z")

</div>

There is also the tar pipe. Piping it to a filtering tool like grep or sed

tar xf -O tarfile.tar.gz | sed ‘regex p’

---

<div class="post-metadata">

**Author:** ![jkingsley](https://ask.cyberinfrastructure.org/letter_avatar_proxy/v4/letter/j/ad7895/32.png) [@jkingsley](https://ask.cyberinfrastructure.org/u/jkingsley)\
**Post date:** [July 6, 2018, 3:23pm UTC](https://ask.cyberinfrastructure.org/t/is-there-a-way-to-process-archived-files-without-fully-extracting-them-to-disk/264/3 "2018-07-06T15:23:36Z")

</div>

Rather than extracting the files, you could consider doing your analysis in a language that supports directly manipulating tar files. For example, python has a `tarfile` module, which has a streaming mode. This will allow you to go through and process your files, without having to ever have them reach a disk.

[Next page](https://ask.cyberinfrastructure.org/t/is-there-a-way-to-process-archived-files-without-fully-extracting-them-to-disk/264.md?page=2)