-
Notifications
You must be signed in to change notification settings - Fork 5
/
Copy pathREADME
258 lines (183 loc) · 8.02 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
#-----------------------------------------------------------------------------#
| UNATTENDED FFMPEG ENCODER |
| https://github.com/supermasita/ufe |
#-----------------------------------------------------------------------------#
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
(!) WORD OF WARNING: This proyect is in a development stage and not even the
README is complete. Use it at your own risk, and if you want, join us :)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
1. INTRODUCTION
1.1 What is UFE ?
1.2 What is not UFE ?
1.3 Features
1.4 To-do
1.5 Credits
2. INSTALLING
2.1 Dependencies
2.2 Creating directories
2.3 Creating database
2.4 Creating FTP
2.5 Configuration
2.6 Installing crons
3. USING UFE
3.1 Encoding workflow
3.1.1 Checker
3.1.2 Encoder
3.1.3 Ftp
4. GETTING HELP
4.1 Troubleshooting
4.2 Known issues
-------------------------------------------------------------------------------
1. INTRODUCTION
1.1 What is UFE?
UFE started as a bunch scripts used to encode and upload video in a
completely unattended way. The goal is to make the encoding and uploading easy
and foolproof, using presets to "normalize" all your videos, providing encoding
best practices and specific quality for each device (broadband, mobile, etc.).
When thinking UFE we had in mind the webmaster (or journalist, etc.)
that has to encode several videos in different qualities for various websites,
and upload them; UFE will make things easier.
You can use UFE as a centralize solution for encoding videos, leaving
the encoding profiles to the Admin and not to the users, disabling them from
FTP access and not hogging their computers with CPU intensive tasks.
UFE could be thought as a simple alternative to other encoding software
such as: Handbrake, Sorenson Squeeze, Windows Movie Maker, etc.
1.2 What is not UFE?
UFE is not a fully featured video edition and encoding software. We
focus on simple and reliable operation.
We plan to extend UFE with other proyects built around it but the "core"
will always be as simple as possible.
1.3 Features
* Multisite in mind.
* Video presets: you can create your own, enable/disable them (globally
or per site).
* Simple MySQL structure (no auto increments, no joins).
* Multiserver in mind (many encoders and master-master DB).
* Uploads to FTP server using year, month, date folder structure.
* Maximum number of FFMPEG processes may be configured for each encoder
(default: 2).
* Non-blocking queues won't let a huge video consume all your processes
all leave all other waiting.
* Logging in syslog.
* Per video FFMPEG log.
* Video hinting.
1.4 To-do
* Create video thumbnail and store it as a blob.
* Weighted queues.
* Handle exceptions.
* Create install script.
* "Pythonize" scripts.
* Create python daemon and avoid crons.
* Monitor file last acces with inotify.
* Improve logging.
* Upload to folder as alternative to FTP.
1.5 Credits
* Alejandro Ferrari - Sysadmin and Wowza expert
* Supermasita - Python coder wannabe
-------------------------------------------------------------------------------
2. INSTALLING
(!) Don't hesitate to search for our help at Github if you have any trouble or
doubt about installation.
2.1 Dependencies
* "FFMPEG" (http://ffmpeg.org/) (if using RHEL best install it from
http://atrpms.net/)
* "Medianfo CLI" (http://mediainfo.sourceforge.net/)
* "Pymediainfo" (https://github.com/paltman/pymediainfo)
* "qt-faststart" (https://github.com/danielgtaylor/qtfaststart)
2.2 Creating directories
1- Create a directory for the scripts (ex: /var/www/html/ufe/), where you will
do the "git clone" or untar.
2- Inside this directory create the following ones :
* "video_origin" : were UFE retrieves videos for encoding.
* "video_original" : where UFE temporaly moves videos retrieved
from "video_origin".
* "video_encoded" : where UFE temporaly stores encoded videos.
3- Inside "video_origin" directory, create another directory called
"default". Here you could create the origin directories for as many
sites as you want to define.
2.3 Creating database
1- Create a database called "ufe" (or whatever you like).
2- Create a user ("ufe"?) and grants to use the database.
3- Use the dump located in "lib" to create the tables, like this
(from command line):
[you@you ufe]$ mysql -u ufe -D ufe -p < lib/database_structure.sql
(!) USE THE DB, USER AND PASS THAT YOU CREATED IN STEP 1 AND 2
2.4 Creating FTP
You will need at least one FTP account to upload your videos. Installing
and configuring an FTP server exceeds the reach of this document, please refer
to any online sources (http://www.cyberciti.biz/ offers great guide).
2.5 Configuration
(!) This is the basic configuration to get you running with the "default" site.
Many sites can be configured in the same fashion. We will extend this README.
1- Connect to DB and you would have to update the following fields in the
"sites" table with your credentials:
* ftp_user
* ftp_pass
* ftp_host
2- Modify the file "lib/config.py", changing the following variables
with your data (examples are provided in the file):
* core_root
* incoming
* original
* encoded
* tmppath
* db_host
* db_user
* db_pass
* db_database
* ffmpeg_bin
2.6 Installing crons
You will need the following crons running (change accordingly to your
needs and setup):
# Main scripts
* * * * * python /var/www/html/ufe/ufe-checker.py > /var/tmp/ufe-checker.log 2>&1
* * * * * python /var/www/html/ufe/ufe-encode.py > /var/tmp/ufe-encode.log 2>&1
* * * * * python /var/www/html/ufe/ufe-ftp.py > /var/tmp/ufe-ftp.log 2>&1
# Recycling (cleaning) of UFE
* * * * * python /var/www/html/ufe/ufe-recycle_encoded.py > /var/tmp/ufe-recycle_encoded.log 2>&1
* * * * * python /var/www/html/ufe/ufe-recycle_original.py > /var/tmp/ufe-recycle_original.log 2>&1
* * * * * python /var/www/html/ufe/ufe-remove_old_registers.py >/var/tmp/ufe-remove_old_registers.log 2>&1
# Cleans .xml and .err files created by "pymediainfo"
* * * * * find /tmp/*.xml -mmin +5 -ls -delete > /var/tmp/ufe-mediainfo_expire_xml.log
* * * * * find /tmp/*.err -mmin +5 -ls -delete > /var/tmp/ufe-mediainfo_expire_err.log
(!) Contact us if you want to use inotify to monitor last access of
files with INCRON:
(http://inotify.aiken.cz/?section=incron&page=why&lang=en)
-------------------------------------------------------------------------------
3. USING UFE
3.1 Encoding workflow
* We assume that a video "test.flv" has been copied inside the
directory "video_original/default".
* "ufe-checker" will scan the directory and if it finds that the file
has not been modified in a given time period (defined in "config.py")
and continue.
* "Mediainfo" will be used to read the video metadata. The file will be
move to "video_original" folder and a register in "video_original"
table will be created.
* Registers will be created in "video_encoded" table for each
"video_profile" defined in "video_profile" table that is globally
enabled, matches the video aspect ratio and is enabled for the site.
* "ufe-encode" is spawned and it will encode any pending video.
* After each encoding, "ufe-ftp" is spawned and any pending upload
video is uploaded.
* "ufe-recycle_encoded" will erase encoded videos that have already
been successfully uploaded, in other to free space.
* "ufe-recycle_original" will erase original videos, whose encoded
videos have been already recycled.
* "ufe-recyle_old_registers" will erase registers for those videos
whose encoded and original videos have been recycled more than
48 hours ago.
3.1.1 Checker (to do)
3.1.2 Encoder (to do)
3.1.3 Ftp (to do)
3.1.4 States (to do)
0 : disabled
1 : pending
2 : running
3 : finished
4 : error
-------------------------------------------------------------------------------
4. GETTING HELP
(!) To be done! Please contact us at Github!
4.1 Troubleshooting
4.2 Known issues