뉴론은 무엇인가
뉴론은 KISTI에서 운영하는 대한민국의 슈퍼컴퓨터로, 대규모의 과학적 계산과 복잡한 데이터 분석 작업을 지원하기 위해 설계되었습니다. 2018년에 도입된 이 슈퍼컴퓨터는 고성능 연산을 위해 최첨단 기술을 통합하고 있습니다. 저는 로그 분석을 통한 보안 위험도 예측 AI 경진대회의 학습을 목적으로 사용하였습니다.
https://dacon.io/competitions/official/235717/overview/description
로그 분석을 통한 보안 위험도 예측 AI 경진대회 - DACON
분석시각화 대회 코드 공유 게시물은 내용 확인 후 좋아요(투표) 가능합니다.
dacon.io
Conda 환경 구축
Conda 환경 구축은 아래의 KISTI의 뉴론 지침서를 참고하여 진행하였습니다.
https://docs-ksc.gitbook.io/neuron-user-guide/undefined-2/appendix-2-how-to-use-conda
Conda | 뉴론 지침서
2024년 3월 21일에 마지막으로 업데이트되었습니다.
docs-ksc.gitbook.io
1. Conda Environment 생성
※ conda initialize를 하지 않았다면 최초 1회 실행
$ conda create -n baseline python=3.9.5
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 23.3.1
latest version: 24.3.0
Please update conda by running
$ conda update -n base -c defaults conda
Or to minimize the number of packages updated during conda update use
conda install conda=24.3.0
## Package Plan ##
environment location: /scratch/@@@@@@@@/.conda/envs/baseline
added / updated specs:
- python=3.9.5
The following NEW packages will be INSTALLED:
_libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main
_openmp_mutex pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu
ca-certificates pkgs/main/linux-64::ca-certificates-2024.3.11-h06a4308_0
ld_impl_linux-64 pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1
libffi pkgs/main/linux-64::libffi-3.3-he6710b0_2
libgcc-ng pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1
libgomp pkgs/main/linux-64::libgomp-11.2.0-h1234567_1
libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1
ncurses pkgs/main/linux-64::ncurses-6.4-h6a678d5_0
openssl pkgs/main/linux-64::openssl-1.1.1w-h7f8727e_0
pip pkgs/main/linux-64::pip-23.3.1-py39h06a4308_0
python pkgs/main/linux-64::python-3.9.5-h12debd9_4
readline pkgs/main/linux-64::readline-8.2-h5eee18b_0
setuptools pkgs/main/linux-64::setuptools-68.2.2-py39h06a4308_0
sqlite pkgs/main/linux-64::sqlite-3.41.2-h5eee18b_0
tk pkgs/main/linux-64::tk-8.6.12-h1ccaba5_0
tzdata pkgs/main/noarch::tzdata-2024a-h04d1e81_0
wheel pkgs/main/linux-64::wheel-0.41.2-py39h06a4308_0
xz pkgs/main/linux-64::xz-5.4.6-h5eee18b_0
zlib pkgs/main/linux-64::zlib-1.2.13-h5eee18b_0
Proceed ([y]/n)? y
Downloading and Extracting Packages
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate baseline
#
# To deactivate an active environment, use
#
# $ conda deactivate
# 환경 활성화
$ conda activate baseline
(baseline) $
2. Conda Environment에 패키지 설치 및 확인
# 필요 패키지 설치
(baseline) $ conda install numpy pandas matplotlib tqdm scikit-learn tensorflow
```
```
(baseline) $ python -c "import sklearn"
(baseline) $
3. Conda Environment 목록 확인
# 설치된 패키지 확인
(baseline) $ conda list
# packages in environment at /scratch/@@@@@@@@/.conda/envs/baseline:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
_tflow_select 2.3.0 mkl
abseil-cpp 20211102.0 hd4dd3e8_0
absl-py 1.4.0 py39h06a4308_0
aiohttp 3.9.3 py39h5eee18b_0
aiosignal 1.2.0 pyhd3eb1b0_0
astunparse 1.6.3 py_0
async-timeout 4.0.3 py39h06a4308_0
attrs 23.1.0 py39h06a4308_0
blas 1.0 openblas
blinker 1.6.2 py39h06a4308_0
bottleneck 1.3.7 py39ha9d4c09_0
brotli 1.0.9 h5eee18b_7
brotli-bin 1.0.9 h5eee18b_7
c-ares 1.19.1 h5eee18b_0
ca-certificates 2024.3.11 h06a4308_0
cachetools 4.2.2 pyhd3eb1b0_0
certifi 2024.2.2 py39h06a4308_0
cffi 1.15.1 py39h74dc2b5_0
charset-normalizer 2.0.4 pyhd3eb1b0_0
click 8.1.7 py39h06a4308_0
conda-pack 0.7.1 py39h06a4308_0
contourpy 1.2.0 py39hdb19cb5_0
cryptography 41.0.3 py39h130f0dd_0
cycler 0.11.0 pyhd3eb1b0_0
cyrus-sasl 2.1.28 h9c0eb46_1
dbus 1.13.18 hb2f20db_0
expat 2.6.2 h6a678d5_0
flatbuffers 2.0.0 h2531618_0
fontconfig 2.14.1 h4c34cd2_2
fonttools 4.51.0 py39h5eee18b_0
freetype 2.12.1 h4a9f257_0
frozenlist 1.4.0 py39h5eee18b_0
gast 0.4.0 pyhd3eb1b0_0
giflib 5.2.1 h5eee18b_3
glib 2.69.1 h4ff587b_1
google-auth 2.6.0 pyhd3eb1b0_0
google-auth-oauthlib 0.4.4 pyhd3eb1b0_0
google-pasta 0.2.0 pyhd3eb1b0_0
grpc-cpp 1.46.1 h33aed49_1
grpcio 1.42.0 py39hce63b2e_0
gst-plugins-base 1.14.1 h6a678d5_1
gstreamer 1.14.1 h5eee18b_1
h5py 3.9.0 py39he06866b_0
hdf5 1.12.1 h70be1eb_2
icu 58.2 he6710b0_3
idna 3.4 py39h06a4308_0
importlib-metadata 7.0.1 py39h06a4308_0
importlib_resources 6.1.1 py39h06a4308_1
joblib 1.2.0 py39h06a4308_0
jpeg 9e h5eee18b_1
keras 2.11.0 py39h06a4308_0
keras-preprocessing 1.1.2 pyhd3eb1b0_0
kiwisolver 1.4.4 py39h6a678d5_0
krb5 1.20.1 h568e23c_1
lcms2 2.12 h3be6417_0
ld_impl_linux-64 2.38 h1181459_1
lerc 3.0 h295c915_0
libbrotlicommon 1.0.9 h5eee18b_7
libbrotlidec 1.0.9 h5eee18b_7
libbrotlienc 1.0.9 h5eee18b_7
libclang 14.0.6 default_hc6dbbc7_1
libclang13 14.0.6 default_he11475f_1
libcups 2.4.2 ha637b67_0
libcurl 8.2.1 h91b91d3_0
libdeflate 1.17 h5eee18b_1
libedit 3.1.20230828 h5eee18b_0
libev 4.33 h7f8727e_1
libffi 3.3 he6710b0_2
libgcc-ng 11.2.0 h1234567_1
libgfortran-ng 11.2.0 h00389a5_1
libgfortran5 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libllvm14 14.0.6 hef93074_0
libnghttp2 1.52.0 ha637b67_1
libopenblas 0.3.21 h043d6bf_0
libpng 1.6.39 h5eee18b_0
libpq 12.15 h37d81fd_1
libprotobuf 3.20.3 he621ea3_0
libssh2 1.10.0 h37d81fd_2
libstdcxx-ng 11.2.0 h1234567_1
libtiff 4.5.1 h6a678d5_0
libuuid 1.41.5 h5eee18b_0
libwebp-base 1.3.2 h5eee18b_0
libxcb 1.15 h7f8727e_0
libxkbcommon 1.0.1 h5eee18b_1
libxml2 2.10.4 hcbfbd50_0
lz4-c 1.9.4 h6a678d5_0
markdown 3.4.1 py39h06a4308_0
markupsafe 2.1.3 py39h5eee18b_0
matplotlib 3.8.4 py39h06a4308_0
matplotlib-base 3.8.4 py39h1128e8f_0
multidict 6.0.4 py39h5eee18b_0
mysql 5.7.24 he378463_2
ncurses 6.4 h6a678d5_0
numexpr 2.8.7 py39h286c3b5_0
numpy 1.26.4 py39heeff2f4_0
numpy-base 1.26.4 py39h8a23956_0
oauthlib 3.2.2 py39h06a4308_0
openjpeg 2.4.0 h3ad879b_0
openssl 1.1.1w h7f8727e_0
opt_einsum 3.3.0 pyhd3eb1b0_1
packaging 23.2 py39h06a4308_0
pandas 2.2.1 py39h6a678d5_0
pcre 8.45 h295c915_0
pillow 10.2.0 py39h5eee18b_0
pip 23.3.1 py39h06a4308_0
ply 3.11 py39h06a4308_0
protobuf 3.20.3 py39h6a678d5_0
pyasn1 0.4.8 pyhd3eb1b0_0
pyasn1-modules 0.2.8 py_0
pycparser 2.21 pyhd3eb1b0_0
pyjwt 2.4.0 py39h06a4308_0
pyopenssl 23.2.0 py39h06a4308_0
pyparsing 3.0.9 py39h06a4308_0
pyqt 5.15.10 py39h6a678d5_0
pyqt5-sip 12.13.0 py39h5eee18b_0
python 3.9.5 h12debd9_4
python-dateutil 2.8.2 pyhd3eb1b0_0
python-flatbuffers 2.0 pyhd3eb1b0_0
python-tzdata 2023.3 pyhd3eb1b0_0
pytz 2023.3.post1 py39h06a4308_0
qt-main 5.15.2 h5b8104b_9
re2 2022.04.01 h295c915_0
readline 8.2 h5eee18b_0
requests 2.31.0 py39h06a4308_1
requests-oauthlib 1.3.0 py_0
rsa 4.7.2 pyhd3eb1b0_1
scikit-learn 1.3.0 py39h1128e8f_1
scipy 1.12.0 py39heeff2f4_0
setuptools 68.2.2 py39h06a4308_0
sip 6.7.12 py39h6a678d5_0
six 1.16.0 pyhd3eb1b0_1
snappy 1.1.10 h6a678d5_1
sqlite 3.41.2 h5eee18b_0
tensorboard 2.11.0 py39h06a4308_0
tensorboard-data-server 0.6.1 py39h52d8a92_0
tensorboard-plugin-wit 1.8.1 py39h06a4308_0
tensorflow 2.11.0 mkl_py39hea27e05_0
tensorflow-base 2.11.0 mkl_py39he5f8e37_0
tensorflow-estimator 2.11.0 py39h06a4308_0
termcolor 2.1.0 py39h06a4308_0
threadpoolctl 2.2.0 pyh0d69192_0
tk 8.6.12 h1ccaba5_0
tomli 2.0.1 py39h06a4308_0
tornado 6.3.3 py39h5eee18b_0
tqdm 4.65.0 py39hb070fc8_0
typing_extensions 4.9.0 py39h06a4308_1
tzdata 2024a h04d1e81_0
unicodedata2 15.1.0 py39h5eee18b_0
urllib3 2.1.0 py39h06a4308_0
werkzeug 2.3.8 py39h06a4308_0
wheel 0.41.2 py39h06a4308_0
wrapt 1.14.1 py39h5eee18b_0
xz 5.4.6 h5eee18b_0
yarl 1.9.3 py39h5eee18b_0
zipp 3.17.0 py39h06a4308_0
zlib 1.2.13 h5eee18b_0
zstd 1.5.5 hc292b87_0
스케줄러(SLURM)를 통한 작업 실행
자세한 내용은 아래 링크를 참고하세요.
https://docs-ksc.gitbook.io/neuron-user-guide/undefined/running-jobs-through-scheduler-slurm
스케줄러(SLURM)를 통한 작업 실행 | 뉴론 지침서
2023년 2월 22일에 마지막으로 업데이트되었습니다.
docs-ksc.gitbook.io
#!/bin/bash
#SBATCH -J baseline
#SBATCH -p skl
#SBATCH --comment python
#SBATCH -o %x_%j.out
#SBATCH -e %x_%j.err
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=20
#SBATCH --time=01:00:00
#SBATCH --mail-type=NONE
export OMP_NUM_THREADS=1
module purge
module load python/3.7.1
source ~/.bashrc
conda activate baseline
python dacon_basic_baseline.py
exit 0