{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Upload benchmarking data to S3 with Neuroglancer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from brainlit.utils import upload\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Uploading Benchmarking Images from local data locations .\n",
    "This notebook demonstrates uploading the benchmarking data and associated `.swc` segment files.\n",
    "The upload destination could easily be set to a url of a cloud data server such as s3.\n",
    "\n",
    "### 1) Define variables.\n",
    " - `source` is the root directory of the data and swc files.\n",
    "     - the .tif file is in the root directory and .swc files are in a subfolder called “consensus-swcs”\n",
    " - `p` is the prefix string. `file://` indicates a filepath, while `s3://` or `gc://` indicate URLs.\n",
    " - `dest` and `dest_segments` are the destinations for the uploads (in this case, filepaths).\n",
    " \n",
    "The below paths lead to sample data in my local drive. Alter the below path definitions to point to your own local file locations.\n",
    "\n",
    "### Note:\n",
    "The below upload destination points to the open-neurodata S3. Uploading data will overwrite the current benchmarking data on S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "source = (Path().resolve().parents[5] / \"Downloads\" / \"validation_21\").as_posix()\n",
    "dest = \"s3://open-neurodata/brainlit/benchmarking_data/validation_21\"\n",
    "dest_segments = \"s3://open-neurodata/brainlit/benchmarking_data/validation_21\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Upload the segmentation data (.swc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "upload.upload_segments(source, dest_segments, 1, benchmarking=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) Upload the image data (.tif)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "upload.upload_volumes(source, dest, 1, benchmarking=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Appendix\n",
    "##### Code for uploading all benchmarking data at once\n",
    "\n",
    "- If when downloading, you get a reshape error, try uploading the segments before uploading the volumes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_list = []\n",
    "validation_list = []\n",
    "for i in range(25):\n",
    "    test_list.append(\"test_\" + str(i + 1))\n",
    "    validation_list.append(\"validation_\" + str(i + 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_res = 1\n",
    "\n",
    "for test in test_list:\n",
    "    print(test)\n",
    "    source = (Path().resolve().parents[5] / \"Downloads\" / test).as_posix()\n",
    "    dest = \"s3://open-neurodata/brainlit/benchmarking_data/\" + test\n",
    "    dest_segments = \"s3://open-neurodata/brainlit/benchmarking_data/\" + test\n",
    "    upload.upload_segments(source, dest_segments, num_res, benchmarking=True)\n",
    "    upload.upload_volumes(source, dest, num_res, benchmarking=True)\n",
    "\n",
    "for val in validation_list:\n",
    "    print(val)\n",
    "    source = (Path().resolve().parents[5] / \"Downloads\" / val).as_posix()\n",
    "    dest = \"s3://open-neurodata/brainlit/benchmarking_data/\" + val\n",
    "    dest_segments = \"s3://open-neurodata/brainlit/benchmarking_data/\" + val\n",
    "    upload.upload_segments(source, dest_segments, num_res, benchmarking=True)\n",
    "    upload.upload_volumes(source, dest, num_res, benchmarking=True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}